# Recognizing the same scene

I have two pictures of a similar scene, and want to recognize whether it is the same scene, photographed in different times and possibly from slightly different angle. For instance, whether it is the same beach (maybe photographed from different angles and at different times) or two different beaches? Is it the same place at a city (again, angles and times are different), or two different places? We are inside a restaurant. Is it the same restaurant, or two different restaurants?

Maybe I have not just two pictures, but two sets of pictures.

Which models, or tools, exist for such a task? How would you attack the problem?

• You want to use recurrent neural networks such as LTSM neural networks – Jon Mar 5 '17 at 23:45

It seems like you're solving a "Manifold Learning" problem (see this overview), where every class/cluster(scene) lies on its own hyper-surface formed by taking photos "in different times and possibly from slightly different angle". Your problem could be significantly simplified with assumption that background doesn't change much when a photo is taken from different angles (or even at different time assuming contrast-invariance), so SFA (Slow Feature Analysis) might be worth to try for a preprocessing in that case.

The overall preferred approach depends on:

• how much data do you have?
• is it possible to get manual labels?
• what is your target accuracy?

So first thing to do is estimation: computational resources, nature and potential sources of data, requirements. For instance, you should estimate an acceptable variation by deciding how many classes of scenes is enough (trade-off between noise robustness and sensitivity). Then you can start to build an initial pipeline for prototyping. Talking about pipeline, if you choose deep-networks as a part of it on some stage of development - the first architecture to try is CNN (see my Note3, 4).

If you don't have any labels in your training set, I'd start with unsupervised learning algorithms for clustering - and try to arrange pictures from same places into same clusters (see this); so if pictures are in the same cluster then it's probably a same scene. I'd try something simple like k-means first (with contrast normalization and maybe some compression, like PCA/SVD) and try to debug it in order to learn more about your data. If it doesn't work or generalize well enough, I'd use some advanced feature extractors like Autoencoders, before passing it to a clustering.

One important point here (especially if you don't have enough training data) is to generate many artificially modified examples - maybe randomly cut some pieces and use compression or denoising autoencoders in order to catch invariant features. It will significantly improve both supervised and unsupervised models of your data. As a side-effect - you'll get some partial labels by assigning same label to artificial samples derived from same image.

On the other hand, if it's easier to manually label your data (in many cases it actually is; note that you don't have to extract a meaningful name of a place) with something like one-hot placeId - you might benefit from supervised learning more (sometimes, a simple classifier is enough), and it would be much simpler. You could use some unsupervised models to get initial labels - and correct them manually.

Note1. It almost always best to apply contrast normalization to your images as a fist stage in pipeline (for both supervised and unsupervised models): https://en.wikipedia.org/wiki/Normalization_(image_processing). You might find that you need more preprocessing after initial debug - it's hard to say what you need if you didn't try anything. Keep in mind that for supervised learning you might not always need PCA compression especially if you have a lot of computational resources - so it's better check if it really improves training performance; besides, PCA is usually redundant for auto-encoders (one-layer auto-encoder gives equivalent results to PCA), but necessary for k-means.

Note1a. Some other preprocessing that might be useful to you is Background substraction - you can try to apply it after contrast normalization stage.

Note2. In most cases you'll have to normalize the size of images too (if you assume that a photo of a same place could be taken by different photographs with different resolution).

Note3. Autoencoders and classifiers (in general sense) are compatible with deep networks (like CNN) - so you can increase the number of layers in order to improve model's capacity. Recommendations here are same as for any MLP. Example: Deeplearning4j MNIST

Note4. If you decide to use CNN (after trying something simpler), there is some important points:

• parameter sharing needs careful management. For instance, if you have some selfie of a person (90 % of the image) with some background (10%), CNN might just ignore that background (your scene) and learn features about person him/herself.
• if your dataset is large enough - try to run it without pooling, as pooling being a form of regularization might dramatically reduce your model's capacity.

P.S. However, your task looks simpler than let's say "Scene Labeling" (you don't need to extract abstract categories), where CNNs are quite popular (see this paper).

Note5. If you choose MLP like CNN, RNN (yes, sometimes they're used for images too), deep-autoencoders and so on, I'd read some book (like DeepLearning by Goodfellow et al.) first - they describe basic recommended "state-of-the-art" approaches for MLP, like using ReLU (instead of sigmoids) for hidden units, advanced regularization methods (dropout, ensembles, pooling and so on), advanced representation learning (feature extraction). Keep in mind, that given enough (Big) data most of algorithms perform very close to each other, so it becomes a matter of computational resources that is available to you.

Note6. Debugging. Andrew Ng's recommendations, IMHO, are best - you can find them in a book mentioned in Note5, Coursera, or just google it.

P.S. As of instruments: R, Octave, Python, TensorFlow (C++/Python), Deeplearning4j (Java) - all support models mentioned above and it's easy to google it. R and Octave are best for initial prototyping.

Example of supervised classification: https://github.com/tensorflow/models/tree/master/inception

Unsupervised example: https://stackoverflow.com/questions/34809795/tensorflow-return-similar-images

• "If you don't have any labels in your training set, I'd start with unsupervised learning algorithms for clustering " - You suggest to cluster images based on raw data? Really? – user31264 Feb 16 '17 at 10:24
• "I'd use some advanced feature extractors", the next line – dk14 Feb 16 '17 at 10:24
• "On the other hand, if it's easier to manually label your data (in many cases it actually is) with something like placeId - you might benefit from supervised learning more (it's just a classifier)" - you suggest to classify images based on raw data??? – user31264 Feb 16 '17 at 10:25
• "Some advantage feature extractors like PCA or Autoencoders" - so you propose to give raw data to PCA, and then give it to cluster analysis or linear classifier? – user31264 Feb 16 '17 at 10:26
• yes. And in case of supervised learning - you can try deep models; the solution basically depends on the target accuracy you're looking for - PCA is just a first prototyping step. – dk14 Feb 16 '17 at 10:31