Hashtags generation and image QA with Watson AI

I've been using Instagram for some years, mainly to showcase my running workouts and travel photos. Each time I copy and paste my usual hashtags from Google Keep, a question keeps popping into my mind: "Why can't Instagram suggest hashtags for me?" With this in mind, I turned to IBM Watson Visual Recognition — a service that uses machine learning to understand the contents of images — to see if I could use it to "read" my photos and suggest hashtags.

To start, I pasted the URL of one of my pictures, showcasing the beautiful Belém Tower, into the service demo page. The general model correctly identified terms that could be suggested as hashtags, but I wanted it to be more specific — could I teach it to properly identify Belém Tower or other monuments?

Training a custom model

I trained Visual Recognition to identify Belém Tower and Pena Palace — two of the most iconic Portuguese monuments — by creating a class for each and collecting 30 example images per class (IBM recommends hundreds or thousands for production use, but 30 was enough for this proof of concept). One misconception I had was that visual recognition needed high-resolution images — in fact, images as low as 320px wide are enough, which also makes training faster.

After zipping the images per class and uploading them, training the model took about 10 minutes. Testing it against new photos, the results were great — Visual Recognition successfully identified Belém Tower and Pena Palace, with a confidence score between 0 and 1, and a default threshold of 0.5 for returning a match.

Auto-hashtagging with PHP

With the monuments model working, I set up the service with a few lines of PHP, using two classifier ids to get both the default Visual Recognition evaluation and my custom monuments model:

<?php

$api_url = 'https://gateway.watsonplatform.net/visual-recognition/api/v3/classify?version=2018-03-19';
$api_key = '{your Visual Recognition API key}';
$image_url = '{the URL of the image to analyse}';
$query = array(
	'url' => $image_url,
	'classifier_ids' => 'default,MonumentsModel_267905574' //insert here your classifiers Ids
);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $api_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1); //POST
curl_setopt($ch, CURLOPT_USERPWD, 'apikey:'.$api_key);
curl_setopt($ch, CURLOPT_POSTFIELDS, $query);
$result = curl_exec($ch);
curl_close($ch);

$array = json_decode($result,true);
$alt = '';
$classScore = '';
$hashtags = '';

for($n=0;$n<2;$n++){
	foreach($array[images][0][classifiers][$n][classes] as $classes){
	  $alt .= ($classes['class'].', ');
	  $classScore .= $classes['class'].':'.number_format($classes['score'],2).'</br>';
	  $hashtags .= '#'.str_replace(' ', '', $classes['class']).' ';
	}
}
?>

<style>
    body{font-family:Helvetica,Arial,sans-serif;font-size:14px;}
    td{border: 1px solid grey;padding:10px;}
</style>
<table>
    <tbody>
        <tr>
            <td rowspan="0"><img width="100%" height="100%" src="<?php echo $image_url; ?>" alt="<?php echo substr($alt, 0, -2); ?>"/></td>
            <td style="vertical-align:top;text-align:right">
                <?php echo $classScore; ?>
            </td>
        </tr>
        <tr>
            <td style="width:250px;color: #003569;">
                <?php echo $hashtags; ?>
            </td>
        </tr>
    </tbody>
</table>

All terms with a minimum score of 0.50 are presented together with my custom model, which clearly identified Belém Tower — shown below as hashtags the user could accept or reject. The same terms can also populate the image's alt tag, useful for screen-reader users.

The Continente Online use case

Our team has a commitment to quality and does its best to provide quality images for each product at Continente Online, but managing a catalog with tens of thousands of products isn't an easy task. In rare cases, a product image doesn't comply with quality standards — dark background, shadows, pixelation, or a missing image — and that's where Watson Visual Recognition plays its role.

I identified the image-quality variants and gathered 30+ examples per class, again using perfect images as the negative class. The results were outstanding — Visual Recognition successfully recognized low-quality images with high confidence, even the placeholder logo image shown when a product has no photo, letting the team detect these situations far more efficiently.

Final thoughts

The setup is quite simple, and with the default models you get results instantly, in multiple languages. Even when a custom model is needed, as this example shows, the process isn't complicated. Give it a try and send me your feedback.