I have a DNN trained on MNIST data (image 1 for digit '4') that recognizes images from the test set with high precision. Each digit is centered and all of them are roughly of the equal size. Will it also recognize image of the same, but transformed (smaller) digit (image 2), same digit in a different location (image 3) and multiple images (image 4)? The size of the input space is the same (e.g. 28x28 pixels).
I think, in general, the answer is no. The input in the neural network is intensity of every pixel, so during training, as the hidden layers learn the features, the weights between the input layer and the first hidden layer are updated based on the value of the particular pixel (think backprop equation), so, for example, the features of the digit '4' will depend on the intensity of the pixels are are most active when this observation is given to the DNN.
Hence, in case of image 3, the activated pixels are very far from those trained with MNIST and they will active completely different neurons, that will combine into unrelated features (if anything sensible at all) and badly misclassify. Something similar should happen in case of image 3. In other words, during testing the features that are associated with this specific digit ('4') will stay quite inactive. If the observation is reasonably close to the pixels of observations in the dataset (image 2), then it can accidentally trigger features from other digits and misclassify too.
A possible solution, I guess, would be some CNN, but I'm not sure how it will know what to transform (and to what size) and when. Any suggestions?