Can a neural network tell if it has seen an image before? I am not trying to classify the image, I just want to know (with a given confidence) if the net has seen the image before.
I am using neural network instead of just linearly searching a database not only because I expect the neural network to be faster, but also it will be much more robust to image alterations (if any).
How would one train such a network? Normally, we provide the training images and the class but in this case, the classes are seen and not seen, which is paradoxical.

EDIT: Facebook research have done something similar. Checkout the paper.
 A: I suppose you are not interested in only recognizing exactly the same image [1]. Instead, you want to know if an image is extremely similar to one you have already seen, and if so, retrieve that one.
Why is that challenging? 
Let us first take a step back and assume that we were only looking at points in Euclidean space. That would be easy, because we could then check if the Euclidean distance $d(x, y) = \sqrt{\sum_i (x_i - y_i)^2}$ exceeds a threshold $\tau$, which you will tune to match the desired degree of necessary similarity. 
However, the Euclidean distance is not meaningful for images: in high dimensional space, nearly all points have equal distance to each other–something commonly referred to as the curse of dimensionality. 
That is why you need to project the images to some space, where it actually is. The easiest way (from a pragmatic perspective) is to get hold of a state-of-the-art image net classifier and use it as a feature extractor. The activations of the last layer before the classification is done will serve as a feature extractor for that. I.e. $d_f(x, y) = \sqrt{\sum_i (f(x)_i - f(y)_i)^2}$ with $f$ mapping an image to the last layers activations, will give you "meaningful" distances. "Meaningful" here refers to the fact that most humans would agree.
There are ways to train these feature extractors from scratch. Convolutional Generative Adversarial Networks or convolutional variational auto-encoders come to my mind. If you want to know more, let me know.
[1] If so, use a hash table for fast lookup and then compare.
A: You can overfit neural network with training images. If you propagate training image (known images) through the network you should get predicted value very close to the expected. For the main training task you can use convolutional autoencoders. Convolutional autoencoder suppose to learn features specific for the training samples.
A: My guess would be that it should be possible, but that it would require two distinct substructures - one to respond to the image and one to classify as old/new. I would do some reading on models of memory such as the ones you find on this page.
Also, I'm not sure how robust such a model would be.
A: I'm not aware of a paper but I would suggest the following scheme -
1) Take an image input and distort till you are ok with noise
2) Feed both image and its distortions to a siamese network and return True
3) Feed image and all other images and return False
4) Repeat
This will give you a prediction on when the images are the same. You can adjust the distortions in your training data to tune to what level of noise are you ok with practically.
The bad news is that you have to maintain a dictionary of ground truth images.
Let's say you don't want to do that.
So another way is to compress the image. I don't think you even need a neural net for that, anything like SVD/PCA will work. Once compressed, hash it and save that hash somewhere. Most images that are similar should 'hash' to the same thing. A new image should hash to something else.
You can combine with other classifiers that count the number of objects in the picture, etc to create a better hash. But idea is the same.
Let me know if either of these ideas are tenable.
