# does it make sense for non-negative data to subtract the mean and divide by the std dev?

It is a very usual procedure to subtract the mean and divide by the standard deviation in a set of data. If we deal with non-negative data, i.e. image, (in [0,1] or [0,255]), does this procedure make sense? Violating the non-negative constraint, what happens?

I add some further considerations.

Suppose you have an image and you decompose it in a set of overlapping patches. Why should you subtract the mean and divide by the std dev for each patch (violating the non-neg prior)?

This procedure is also used in dictionary learning and sparse coding. In dictionary learning, given an image ($y$), a standard approach is dividing it into a set of patches ($p$), then subtract the mean ($p_m$) and dividing by the std deviation ($p_s$).

Is it a crucial step if data are non-negative?

• Why would it matter, for this process of standardization, if the data is negative or positive? "What happens?" is you standardize the data. Jan 15 '13 at 11:51
• After this standardization, there are pos and neg values. What do neg values represent in an image? If I know that data are non-negative, why should I violate this "prior"? Jan 15 '13 at 12:43
• The standardized variable is not the raw variable so you are not violating any "prior" by having negative values. In fact you're guaranteed to have negative values if the variable has non-zero variance. A more important question is: why are you standardizing? When you standardize, the result is interpreted as the number of standard deviations above/below the mean. That interpretation is particularly meaningful for normally distributed data but non-negative data is quite possibly skewed, so I am curious why you decided to standardized. Is this a predictor in a regression model, or...? Jan 15 '13 at 13:13
• As others have written: it all depends on what you are going to do with the data afterwards (feed it to a regression, ...). That said, my personal first impulse for nonnegative data would by not to subtract the mean and divide by the sd, but to scale it linearly to [0,1]. Jan 15 '13 at 13:26
• One productive guiding idea in data analysis (due to John Tukey) is not to let the form in which you record the data determine how you analyze it. (This is in sharp contradistinction to Stevens' taxonomy of nominal-ordinal-interval-ratio data, which many people interpret as saying that the form dictates or limits the possible analyses.) From Tukey's point of view, then, your question ("does this procedure make sense"?) is excellent and ought to be answered in terms of whether the procedure provides simplification or insight (but non-negativity of the values is of little relevance).
– whuber
Jan 15 '13 at 14:30

First of all, there have been several questions on standardization already, e.g.

Subtracting the mean is one way of centering your data: The average becomes the new origin in the "point cloud description" of the data (each case is a point in $p$ dimensions, for RGB images, $p = 3$). Properly centered data can lead to numerically more stable models, and centering may also help in the interpretation of data and models: it sets a "baseline", and the centered data records deviations from this.
Whether this is a sensible idea depends on your data: for some data it does make sense, for other data another center may be more appropriate, yet other data sets do already have a useful center. E.g. in the example of star photographs, you may want to find out the average background color and subtract that.

Dividing by the standard deviation (or the variance) standardizes the data. This can be useful to achieve equal weights for all input channels in the subsequent data analysis. In other cases, is is not sensible. The latter may very well be the case for your data: your variates already share their physical unit. However, you may want to calibrate them to correct the wavelength dependence of the camera's sensitivity (whitelight correction).

You may also want to adjust all channels together: that would be adjusting contrast and brightness, which are also a way to center and standardize.

Since you mention sparse coding, I assume you are referring to natural images.

For natural images, standardization is often carried out because natural image patches have pretty stable statistical properties once you subtracted the constant part (and whitened them; see below). You may look at it like this: A natural image patch $p$ has a mean lumination (the mean of the patch) and a contrast (the standard deviation of the patch). If you are interested in the content of the patch, then it is a good idea to subtract the mean lumination and divide out the contrast to map all image patches with the same content on the same point.

Natural image patches have pretty stable statistical properties after subtracting the mean (also often called DC component). For reference you could look at papers by David Field, Bruno Olshausen, David Ruderman, Eero Simoncelli, Matthias Bethge, or Aapo Hyvaerinen. Interestingly, the statistics of the DC component varies a lot from image to image (if you sample many patches from one image), but the statistical properties of the patches are quite stable. This is true in particular for whitened patches, i.e. when you divide by the standard deviation in the PCA basis (a whitening matrix is not unique, but the PCA version is one possible choice). Note that many sparse coding models are actually trained on DC-subtracted and whitened natural image patches.

In short: For natural images you like to do the standardization because the probability models fitted to the standardized patches generalize better from image to image.

• Suppose you have a very dark image with only few light points (think about astronomical images). Therefore, even though you could prune black patches (i.e., patches with an intensity mean near to $0$), most of patches would be very dark (excluding patches with the light points). In this context (maybe different from common natural images), it is not clear whether subtracting the mean and dividing by the std dev it would be really useful (and meaningful). Jan 15 '13 at 22:49
• I am not sure whether I agree. Suppose you have two patches containing stars. In one patch they were far away or the atmospheric conditions were suboptimal so the stars are less bright. On the other patch the stars were near and the photo was taken in a clear night, so the image is brighter. You might be interested in normalizing them onto a common scale (I think this is basically what's done in auto-level and auto-contrast functions in Photoshop or other image processing programs). Jan 16 '13 at 12:22