In the MNIST dataset, are the images on white or black background? I seem to have encounter both type of images by googling around. Does the background color has any effects on the performance of a neural network in a recognition task?
The home of the MNIST dataset is at http://yann.lecun.com/exdb/mnist/
The page says: "Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black)."
What matters to a neural net are the pixel values 0 (background) and 255 (foreground), though you can assign different colors to these pixel values when plotting the images.
A neural net trained on images with 0 as background and 255 as foreground will not be able to recognize images with inverted colors. You'd have a train a neural net on both types of images if you want it to work on both.