4
$\begingroup$

I was reading a paper related to Auto encoders for my project work. It is required to input images as vectors to the neural network. I couldn't understand a certain sentence due to lack of knowledge of statistics (I guess). I Googled, but the problem is I don't know what it is exactly and searching the same phrase returns the same kind of documents but not their explanation.

Source: http://www.cs.toronto.edu/~hinton/absps/esann-deep-final.pdf

We train on 1.6 million 32*32 color images that have been preprocessed by subtracting from each pixel its mean value over all images and then dividing by the standard deviation of all pixels over all images.

What does it mean by "subtracting from each pixel its mean value over all images and then dividing by the standard deviation of all pixels over all images".

My interpretation is: "Subtracting from each pixel its mean value over all images" It means, for a pixel position in an image, subtract the average of values of that pixel position over all images and subtract from the current pixel value.

Am I correct?

It is somewhat ambiguous to me.

Please explain in some math terms.

Improve this question
$\endgroup$
5
  • $\begingroup$ @NickCox I've added the link. In first senetence it asked us to subtract mean of each pixel overall images but standard deviation is on over all pixels and all images, so, in SD formula which mean should I use is it mean of that pixel position or mean of all pixels of all images? More importantly, should I take means and sds differently for r,g,b domains or combine rgb as one value and calculate this. $\endgroup$
    – pinkpanther
    Commented Oct 12, 2013 at 11:54
  • $\begingroup$ @NickCox Thank you very much!, if possible consider adding an answer. More importantly, should I take means and sds differently for r,g,b domains or combine rgb as one value and calculate this?. In general what is preferred? $\endgroup$
    – pinkpanther
    Commented Oct 12, 2013 at 12:03
  • $\begingroup$ Glad that helped, but now this is a morphing into a quite different new question in image processing, and (1) you should pose that in a new thread (2) it's not clear to me that it is essentially a statistical question that belongs here (3) sorry, but I am not experienced enough in that field to advise you. $\endgroup$
    – Nick Cox
    Commented Oct 12, 2013 at 12:07
  • $\begingroup$ @NickCox I mean if you don't mind please add answer to this thread question so that I can mark as accepted. I don't need an answer for the question in the comment. Sorry if I'm troubling you. $\endgroup$
    – pinkpanther
    Commented Oct 12, 2013 at 12:11
  • $\begingroup$ OK; I combined my earlier comments into an answer (and deleted the corresponding comments). $\endgroup$
    – Nick Cox
    Commented Oct 12, 2013 at 12:16

1 Answer 1

Reset to default
4
$\begingroup$

Each image is composed of 32 $\times$ 32 pixels, so for a given pixel (say row 13, column 31) something measured is averaged over all the images, and the standard deviation (SD for short) for the same something is also calculated.

(value − mean) / SD is often called a z-score and is a way of standardizing values to take account of mean and SD. Presumably that's done for every pixel, meaning every pixel position.

It is spelled out that they are "dividing by the standard deviation of all pixels over all images" [my emphasis] and that SD would usually be calculated with reference to the corresponding overall mean. However, division by that SD would be dividing by a constant, so it won't have any effect on the images beyond a question of units.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Regarding your last sentence : In the case of neural networks (such as the autoencoders mentionned in the question), the division by a constant may in fact have quite an effect regarding numerical issues and performance. $\endgroup$
    – Soltius
    Commented Apr 16, 2018 at 14:23
  • $\begingroup$ @Soltius I can happily believe you. Thanks! Is this documented or just well known? $\endgroup$
    – Nick Cox
    Commented Apr 16, 2018 at 16:55
  • 1
    $\begingroup$ It is known to have many interesting/necessary effects such as : a/ feature scaling (maybe less relevant in the case of images), b/ avoid saturation of hidden units (can also be expressed as "inputs fall in the useful range of the non-linearities), c/ data standardization which helps your network to generalize to new unseen data since variability is reduced... The "go to" paper is usually LeCun's one yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf (section 4.3) although it does not talk about std specifically. But simply googling "why do we normalize in neural networks" gives a good idea $\endgroup$
    – Soltius
    Commented Apr 17, 2018 at 7:16
  • 1
    $\begingroup$ of the current state of knowledge, including right here on Cross-Validated eg stats.stackexchange.com/questions/7757/… or on SO stackoverflow.com/questions/4674623/…. $\endgroup$
    – Soltius
    Commented Apr 17, 2018 at 7:21
  • 1
    $\begingroup$ Anyway, sorry for necromancing such an old answer, I didn't want to point out any lack of precision on your behalf, just add some info (which probably emerged to the ML comunity somewhere between your answer and now) in case someone comes here today ! $\endgroup$
    – Soltius
    Commented Apr 17, 2018 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.