Image Tagger - Multi-Label CNN Image Classification

Smart Hashtag Generation

In today’s article, we are going to perform Multi-Label Image Classification using Convolutional Neural Networks. Machine Learning model trained with such an approach, would be able to generate multiple descriptive labels for a given input image. Such an application could be used to generate hashtags for social media posts, like in the below example ().

Multi-Class CNN Image Classification

Before we dive into the multi-label classification, let’s start with the multi-class CNN Image Classification, as the underlying concepts are basically the same with only a few subtle differences. If you are completely new to this field, I recommend you start with the following article to learn the basics of this topic.

Multi-Class vs Multi-Label Classification

To learn what is a multi-label classification, let’s use an analogy to the multi-class classification and illustrate it with the following example.

Multi-Class

Multi-Label

What’s the difference?

The difference between classes and labels is that classes are mutually exclusive and labels are not.

It basically means that an instance can belong to only one class, but can have multiple labels.

For example, in the very last example, a happy golden retriever can be labeled as not only a dog, by as an animal as well, giving us an option to assign multiple descriptive labels to a given instance.

Multi-Label CNN Image Classification

Dataset

In order to perform multi-label classification, we need to prepare a valid dataset first. Valid in that case, means that every image has associated multiple labels.

I’ve collected 758901 of 224x224 center-cropped various images of people, animals, places, gathered from unsplash, instagram and flickr.

An example sample looks like the following.

0007ab8c-animals_outdoors_wildlife.jpg
  • There is 1120 unique labels.
  • Every image has 1–10 labels.
  • Every label has at least 100 of the corresponding samples.
  • Image-labels mappings are in the metadata.csv.
  • Besides that, every image follows the following convention: id-tag1_tag2.jpg

Full dataset is available .

Training

Having collected the right data, we can proceed to build our multi-label image classifier a.k.a. smart hashtag generator.

You can find the corresponding code in the below link.

I won’t dive into the details of the training code as it is very similar to the code explained , but there are some subtle differences worth bringing up.

Activation Function of the Final Layer

Knowing that in the multi-class problem classes are exclusive, we may pick a softmax activation function that is guaranteed to output values that sum up to 1.0. However, in the multi-label problem labels are not exclusive so we may choose sigmoid activation instead.

Evaluation Metrics

One of the first choices of evaluation metrics of a multi-class problem is accuracy.

However, in the multi-label problem, where the number of unique labels in total is very high, and a number of labels per instance in a datest is low (like in our application where we have 1120 unique labels, but each image in the dataset has on average only 4 labels), it’s very easy to get very high accuracy simply by predicting 0.0 for majority of the labels.

In such a case, we may need to look into other metrics like precision or recall. High precision means that the model is very accurate in its predictions and high recall means that it doesn’t overlook many samples.

Finally, we can calculate an f1 score which is a harmonic mean over the precision and recall, which tries to find a ‘balance’ between these two.

Results

Below test results with corresponding confidence scores indicate that our model after hours of training on a GPU learned how to assign correct labels to the given images.

What’s Next?

In this tutorial, we’ve learned how to perform a Multi-Label CNN Image Classification. Such knowledge can be used to build or with some additional , it can be used to create automatic image captions. Possibilities are endless and I am looking forward to seeing your projects and results!

Project’s GitHub page:

Corresponding iOS app:

Questions? Comments? Feel free to leave your feedback in the comments section or contact me directly at .

And don’t forget to 👏 if you enjoyed this article 🙂.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store