2 humbly attempted to improve brevity of title

1

Why do we normalize images by subtracting the dataset's image mean and not the current image mean in deep learning?

There are some variations on how to normalize the images but most seem to use these two methods:

1. Subtract the mean per channel calculated over all images (e.g. VGG_ILSVRC_16_layers)
2. Subtract by pixel/channel calculated over all images (e.g. CNN_S, also see Caffe's reference network)

The natural approach would in my mind to normalize each image. An image taken in broad daylight will cause more neurons to fire than a night-time image and while it may inform us of the time we usually care about more interesting features present in the edges etc.

Pierre Sermanet refers in 3.3.3 that local contrast normalization that would be per-image based but I haven't come across this in any of the examples/tutorials that I've seen. I've also seen an interesting Quora question and Xiu-Shen Wei's post but they don't seem to support the two above approaches.

What exactly am I missing? Is this a color normalization issue or is there a paper that actually explain why so many use this approach?