# Can a neural network tell if it has seen an image before?

I am not trying to classify the image, I just want to know (with a given confidence) if the net has seen the image before.

I am using neural network instead of just linearly searching a database not only because I expect the neural network to be faster, but also it will be much more robust to image alterations (if any).

How would one train such a network? Normally, we provide the training images and the class but in this case, the classes are seen and not seen, which is paradoxical.

• I don't see how you could train such a network. If the output is "1 = seen, 0 = new", then any image in the training set will have 1 as target output - so the network would just learn to set the output to 1, always. – nikie Dec 4 '16 at 17:35
• @nikie I imagine that you could have one network that would rapidly store information when presented with an image in such a way that new images produce low coactivity in network nodes while old images would produce higher coactivity. You would then have a second structure that would respond to activity in the first network on the basis of the coactivity level - something like a signal detection approach. Ultimately, the fact that the human brain can do it (which I see as a network) suggests there must be some mechanism for accomplishing this. May not be easy, though. – J Taylor Dec 4 '16 at 17:49
• @nikie Also, it's been a while since I looked at it, but I feel like the Minerva2 model by Hintzman (referenced on the linked page in my answer) provides a prototype of how recognition memory could be made to work in this way. – J Taylor Dec 4 '16 at 17:53
• What purpose does it serve to store memory? If you have a model in production that sees thousands of images daily, this would cause some storage/memory issues. – Jon Dec 4 '16 at 19:15

I suppose you are not interested in only recognizing exactly the same image [1]. Instead, you want to know if an image is extremely similar to one you have already seen, and if so, retrieve that one.

Why is that challenging?

Let us first take a step back and assume that we were only looking at points in Euclidean space. That would be easy, because we could then check if the Euclidean distance $d(x, y) = \sqrt{\sum_i (x_i - y_i)^2}$ exceeds a threshold $\tau$, which you will tune to match the desired degree of necessary similarity.

However, the Euclidean distance is not meaningful for images: in high dimensional space, nearly all points have equal distance to each other–something commonly referred to as the curse of dimensionality.

That is why you need to project the images to some space, where it actually is. The easiest way (from a pragmatic perspective) is to get hold of a state-of-the-art image net classifier and use it as a feature extractor. The activations of the last layer before the classification is done will serve as a feature extractor for that. I.e. $d_f(x, y) = \sqrt{\sum_i (f(x)_i - f(y)_i)^2}$ with $f$ mapping an image to the last layers activations, will give you "meaningful" distances. "Meaningful" here refers to the fact that most humans would agree.

There are ways to train these feature extractors from scratch. Convolutional Generative Adversarial Networks or convolutional variational auto-encoders come to my mind. If you want to know more, let me know.

[1] If so, use a hash table for fast lookup and then compare.

• I think an autoencoder would be more suitable for the job. We can then compute the euclidean distances between the latent spaces. – Souradeep Nanda Jul 11 '18 at 2:18

You can overfit neural network with training images. If you propagate training image (known images) through the network you should get predicted value very close to the expected. For the main training task you can use convolutional autoencoders. Convolutional autoencoder suppose to learn features specific for the training samples.

• As far as I know convolutional auto-encoders will do fine even if I give it a picture it has not seen before. Many smart de-noising programs use it. – Souradeep Nanda Dec 5 '16 at 5:12
• Yes, but the goal is to build a bad autoencoder instead of a good one. It won't be suitable for image that it hasn't seen before, because it will try to extract features that relevant only for training images. – itdxer Dec 5 '16 at 10:50
• I got another idea. I can over fit some RNN on the sequence of pixels of each image. If the RNN is "surprised" or unable to predict the next pixel then the image is new. If the RNN was able to predict a large number of previous pixels then it has seen it before. However, I doubt it would be practical unless I use some kind of convolutional RNN. – Souradeep Nanda Dec 6 '16 at 2:32

My guess would be that it should be possible, but that it would require two distinct substructures - one to respond to the image and one to classify as old/new. I would do some reading on models of memory such as the ones you find on this page.

Also, I'm not sure how robust such a model would be.

• Very informative, but I have yet to figure out the practicalities. – Souradeep Nanda Dec 6 '16 at 3:37

I'm not aware of a paper but I would suggest the following scheme -

1) Take an image input and distort till you are ok with noise

2) Feed both image and its distortions to a siamese network and return True

3) Feed image and all other images and return False

4) Repeat

This will give you a prediction on when the images are the same. You can adjust the distortions in your training data to tune to what level of noise are you ok with practically.

The bad news is that you have to maintain a dictionary of ground truth images.

Let's say you don't want to do that. So another way is to compress the image. I don't think you even need a neural net for that, anything like SVD/PCA will work. Once compressed, hash it and save that hash somewhere. Most images that are similar should 'hash' to the same thing. A new image should hash to something else.

You can combine with other classifiers that count the number of objects in the picture, etc to create a better hash. But idea is the same.

Let me know if either of these ideas are tenable.

• Interesting, I will try it out. – Souradeep Nanda Dec 15 '16 at 8:26