4
$\begingroup$

I'm working on a multi-class classification problem using CNN. Most of the images of each class have a text/watermark at a specific position on the image.

I have a couple of questions.

  1. Does the watermark/text influence the CNN in feature extraction and then in the classification of the images?

  2. Is it fine to train my CNN model with images containing watermark/text or is it necessary to remove the watermark/text before training the CNN model?

$\endgroup$

1 Answer 1

5
$\begingroup$

Yes, it can be a problem. A very similar example was used in the Unmasking Clever Hans Predictors and Assessing What Machines Really Learn paper by Lapuschkin et al (see below). They show an example of a neural network that learned to detect "horse" based on the fact that training set images with horses contained a textual tag.

Figure 2: Assessing problem-solving capabilities of learning machines using explanation methods. (a) The Fisher
vector classifier trained on the PASCAL VOC 2007 data set focuses on a source tag present in about one fifth of the
horse figures. Removing the tag also removes the ability to classify the picture as a horse. Furthermore, inserting
the tag on a car image changes the classification from car to horse. (b) A neural network learned to play the Atari
Pinball game. The model moves the pinball into a scoring switch four times to activate a multiplier (indicated as
symbols marked in yellow box) and then maneuvers the ball to score infinitely. This is done purely by “nudging the
table” and not by using the flippers. In fact, heatmaps show that the flippers are completely ignored by the model
throughout the entire game, as they are not needed to control the movement of the ball. (c) Development of the
relative relevance of different game objects in Atari Breakout over the training time. Relative relevance is the mean
relevance of pixels belonging to the object (ball, paddle, tunnel) divided by the mean relevance of all pixels in the
frame. Thin lines: 6 training runs. Thick line: average over the 6 runs.

The same might apply to your problem as well: if some of the classes have pictures with watermarks and some don't, the neural network can learn to simply detect the watermark to make the classification. It can also be a problem if you have different watermarks or different placements of watermarks in different classes. But it is not only about watermarks, if you have images from different sources for different classes (or different proportions of them for different classes) the model can learn to detect features of the images. An example of this was presented in another paper, where the model learned to detect a "husky" dog based on the fact alone that the pictures of huskies were all made in winter and contained snow.

Interpretability algorithm marked the parts of image of a husky dog in terms of their importance for making the classification, dog was not important, snow was important.

Unfortunately, removing the watermarks does not necessarily need to help. In the examples used by Lapuschkin et al it did help, but keep in mind that there is no way to perfectly remove the watermark. You may be able to remove the watermark so it is not visible to a human, but in most such cases it is possible to detect the removed watermark. It can be done with neural networks, but also with much more primitive algorithms.

So the watermarks may be a problem and if using this data you should pay great attention to validating if or how your model treats the watermarks on the images (removed or not) and how they influence your results.

Finally, the images are watermarked most likely because you didn't pay for the commercial use license for the images. In such a case, it may be illegal for you to use the images to train the algorithm and you should first consult with a lawyer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.