150
$\begingroup$

There are some variations on how to normalize the images but most seem to use these two methods:

  1. Subtract the mean per channel calculated over all images (e.g. VGG_ILSVRC_16_layers)
  2. Subtract by pixel/channel calculated over all images (e.g. CNN_S, also see Caffe's reference network)

The natural approach would in my mind to normalize each image. An image taken in broad daylight will cause more neurons to fire than a night-time image and while it may inform us of the time we usually care about more interesting features present in the edges etc.

Pierre Sermanet refers in 3.3.3 that local contrast normalization that would be per-image based but I haven't come across this in any of the examples/tutorials that I've seen. I've also seen an interesting Quora question and Xiu-Shen Wei's post but they don't seem to support the two above approaches.

What exactly am I missing? Is this a color normalization issue or is there a paper that actually explain why so many use this approach?

Improve this question
$\endgroup$
4
  • $\begingroup$ I don't know the answer, but have you tried each of the method? Is their any difference in the performances? $\endgroup$
    – user112758
    May 8, 2016 at 12:38
  • $\begingroup$ @user112758 - implementing them is a little painful (especially for the by-pixel) and my experience is that normalizing per image works fine but my data is not that representative. I'll try to experiment with the normalization but I'm curious to hear the motivation behind these (in my mind) strange normalization procedures. $\endgroup$
    – Max Gordon
    May 8, 2016 at 12:41
  • $\begingroup$ Ok, maybe you can ask this in the caffe Google group caffe GitHub issues. I guess there would be more experts on this topic. $\endgroup$
    – user112758
    May 8, 2016 at 12:50
  • $\begingroup$ Probably per-dataset normalization during training means less calculations - you have the mean values, just subtract and go instead of caring for each image. During inference/validation - well, I don't see why, so I'll write some code and test for this. $\endgroup$
    – A. Genchev
    Jan 14, 2021 at 13:28

5 Answers 5

Reset to default
113
$\begingroup$

Subtracting the dataset mean serves to "center" the data. Additionally, you ideally would like to divide by the sttdev of that feature or pixel as well if you want to normalize each feature value to a z-score.

The reason we do both of those things is because in the process of training our network, we're going to be multiplying (weights) and adding to (biases) these initial inputs in order to cause activations that we then backpropogate with the gradients to train the model.

We'd like in this process for each feature to have a similar range so that our gradients don't go out of control (and that we only need one global learning rate multiplier).

Another way you can think about it is deep learning networks traditionally share many parameters - if you didn't scale your inputs in a way that resulted in similarly-ranged feature values (ie: over the whole dataset by subtracting mean) sharing wouldn't happen very easily because to one part of the image weight w is a lot and to another it's too small.

You will see in some CNN models that per-image whitening is used, which is more along the lines of your thinking.

Improve this answer
$\endgroup$
12
  • 8
    $\begingroup$ Thank you for the answer. I'm familiar with the concept of centering the data and making the sure the range is similar in order to get stable gradients. The question is more of why we need to do this over the entire dataset and why this would help in contrast to per-image whitening? I would like a simple reference that shows in some way that this improves learning before I accept the answer. I know that batch normalization is an incredibly powerful technique but I don't see the connection to entire dataset normalization. $\endgroup$
    – Max Gordon
    Jun 28, 2016 at 18:50
  • 4
    $\begingroup$ They are. And you can update however frequently you want - the analytical implications are identical which is what's so nice and scalable about gradient descent. The reason that we use stochastic gradient descent (shuffling input order + batching) is to smooth out our hill climbing through gradient space. Given a single point we can't really be sure our update will push us in the direction of local maxima, however if you select enough points, this likelihood becomes higher (in expectation). $\endgroup$
    – lollercoaster
    Jun 29, 2016 at 17:41
  • 5
    $\begingroup$ How does this help get features into a similar range? If I have two images, one ranging from 0 to 255 and one ranging from 0 to 50 in pixel values, say with a mean of 50 and stdev of 15. Normalizing gives me image 1 ranging from -3.3 to 13.6 and image 2 ranging from -3.3 to 0. They still aren't in the same scale. $\endgroup$
    – Daniel
    Nov 17, 2018 at 14:48
  • 1
    $\begingroup$ @Daniel, not every image after normalizing will be "in the same range", but as a group, a normalized feature's values will. See: en.wikipedia.org/wiki/Standard_score $\endgroup$
    – lollercoaster
    Apr 13, 2020 at 22:07
  • 1
    $\begingroup$ @MaxGordon the reason is simply that the gradient goes too large if features are not normalized using images / 255. I don't know more but that may help. $\endgroup$
    – Minsky
    Jan 15, 2021 at 11:26
11
$\begingroup$

Prior to batch normalization, mean subtraction per channel was used to center the data around zero mean for each channel (R, G, B). This typically helps the network to learn faster since gradients act uniformly for each channel. I suspect if you use batch normalization, the per channel mean subtraction pre-processing step is not really necessary since you are normalizing per mini-batch anyway.

Improve this answer
$\endgroup$
4
  • 9
    $\begingroup$ "I suspect if you use batch normalization, the per channel mean subtraction pre-processing step is not really necessary since you are normalizing per mini-batch anyway." But batch norm refers to normalizing the weights in the layers of your network...not the actual input images. These are 2 different things. $\endgroup$
    – Monica Heddneck
    Feb 6, 2019 at 20:54
  • 3
    $\begingroup$ @MonicaHeddneck I don't believe that is true. Batch normalization, and normalization in general, usually refers to normalizing the outputs (or inputs, if thought of that way) into / out from the network - not the weights of the network. $\endgroup$
    – Dan Nissenbaum
    Jul 18, 2020 at 10:17
  • 3
    $\begingroup$ @DanNissenbaum - there are two definitions of batch norm outside the "white-paper world". However I think the paper that user this technique used it as "normalization between layers": See: "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" arxiv.org/abs/1502.03167 $\endgroup$
    – Mindaugas Bernatavičius
    Aug 31, 2020 at 6:49
  • $\begingroup$ I can't believe a comment incorrectly saying batch norm is used to normalize weights has almost 5 times as many up votes as the comment correcting it with the right info. BN normalizes output/input of a layer not the weights. $\endgroup$
    – Frobot
    Apr 20 at 17:57
8
$\begingroup$

Per-image normalization is common and is even the only in-built function currently in Tensorflow (primarily due to being very easy to implement). It is used for the exact reason you mentioned (day VS night for the same image). However, if you imagine a more ideal scenario where lighting was controlled, then the relative differences between each image would be of great value in the algorithm, and we wouldn't want to wipe that out with per-image normalization (and would want to do normalization in the context of the entire training data set).

Improve this answer
$\endgroup$
7
$\begingroup$

There are two aspects to this topic:

  1. Normalization to keep all data in the same scale --> the outcome is going to be similar when normalizing both on a per-image basis or across the entire image data set
  2. Preservation of relative information --> this is where doing normalization on a per-image or per-set basis makes a big difference

For example if you want to learn a CNN to recognize night scenes vs. daytime scenes and you normalize on a per-image basis, the network will fail miserably because all the images will be scaled equally.

Another pitfall of per-image normalization is that you may be artificially gaining up image sensor shot noise (e.g. for very dark scenes) and this will throw off the CNN in confusing such noise as useful information.

Last word of caution on normalization: if it is done incorrectly it can lead to unrecoverable loss of information, for example image clipping (generating values that are below the valid range of the image datatype) or saturation (above the valid range). This is a classic mistake when operating with uint8 variables to represent images and values either go below 0 or exceed 255 due to normalization / pre-processing operations. Once this happens, image information is lost and it cannot be recovered, so the CNN will fail to learn any useful information from those image pixels.

Improve this answer
$\endgroup$
4
$\begingroup$

This is called preprocessing of data before using it. You can process in many ways but there is one condition that you should process each data with same function X_preproc = f(X) and this f(.) should not depend on data itself, so if you use the current image mean to process this current image then your f(X) will actually be really f(X, image) and you don't want that.

The image contrast normalization you were talking about is for a different purpose. Image contrast normalization will help in feature.

But f(.) above will help on optimization by keeping all the features numerically equal to each other (of-course approximately)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.