Recognition of digit of size other than those in the training set using DNNs

Question

I have a DNN trained on MNIST data (image 1 for digit '4') that recognizes images from the test set with high precision. Each digit is centered and all of them are roughly of the equal size. Will it also recognize image of the same, but transformed (smaller) digit (image 2), same digit in a different location (image 3) and multiple images (image 4)? The size of the input space is the same (e.g. 28x28 pixels).

I think, in general, the answer is no. The input in the neural network is intensity of every pixel, so during training, as the hidden layers learn the features, the weights between the input layer and the first hidden layer are updated based on the value of the particular pixel (think backprop equation), so, for example, the features of the digit '4' will depend on the intensity of the pixels are are most active when this observation is given to the DNN.

Hence, in case of image 3, the activated pixels are very far from those trained with MNIST and they will active completely different neurons, that will combine into unrelated features (if anything sensible at all) and badly misclassify. Something similar should happen in case of image 3. In other words, during testing the features that are associated with this specific digit ('4') will stay quite inactive. If the observation is reasonably close to the pixels of observations in the dataset (image 2), then it can accidentally trigger features from other digits and misclassify too.

A possible solution, I guess, would be some CNN, but I'm not sure how it will know what to transform (and to what size) and when. Any suggestions?

Answer 1

You're right that the answer is no. Using a CNN is one approach - the max-pooling operation makes it a little more translation invariant with each added layer, until in the final layer you just end up with a bag of features.

Another approach, which is a bit more tricky and more recent, is to use attention (a mechanism which focuses on a certain part of the image, then feeds that to a neural net). The attention mechanism can either be trained through reinforcement learning or backpropagation if it is differentiable (the DRAW paper does something like this, but it's not a normal MLP).

If you're actually just working with digits on a blank background, some kind of hand-coded recenter/rescale/deskew system is probably best.

Answer 2

May be you can try generating training dataset with these features and then train on that one. I faced a similar problem where the digit could possibly shift its location which was giving me wierd results , so i manually created a dataset with content shifted from center in all the directions and it worked for me.

I dont know about the double 4 in one image