Why do we divide by the standard deviation and not some other standardizing factor before doing PCA?

Question

I was reading the following justification (from cs229 course notes) on why we divide the raw data by its standard deviate:

even though I understand what the explanation is saying, it is not clear to me why dividing by the standard deviation would achieve such a goal. It says so that everyone is more on the same "scale". However, its not entirely clear why dividing by the standard deviation achieves that. Like, whats wrong with dividing by the variance? Why not some other quantity? Like...the sum of absolute values? or some other norm... Is there a mathematical justification for choose the STD?

Are the claims in this extract a theoretical statement that can be derived/proved through mathematics (and/or statistics) or is it more one of those statement that we do because it seems to work in "practice"?

Basically, can one provide either a rigorous mathematical explanation of why that intuition is true? Or if its just an empirical observation, why we think that works in general before doing PCA?

Also, in the context of PCA, is this the process of standardizing or normalizing?

---

Some other thoughts I had that might "explain" why the STD:

Since PCA can be derived from maximizing the variance, I guessed that dividing by a related quantity such as the STD, might be one of the reasons we divided by the STD. But then I considered that maybe if we defined maybe a "variance" with any other norm, $\frac{1}{n} \sum^{n}_{i=1} (x_i -\mu)^p$, then we would divide by the STD of that norm (by taking the pth root or something). Though, it was just a guess and I am not 100% about this, hence the question. I was wondering if anyone knew anything related to this.

---

I did see that there was maybe a related question:

PCA on correlation or covariance?

but it seemed talk more about when to use "correlation" or "covariance" but lacked rigorous or convincing or detailed justifications, which is what I am mainly interested in.

Same for:

Why do we need to normalize data before analysis

related:

"Normalizing" variables for SVD / PCA

Answer 1

This is in partial answer to "it is not clear to me why dividing by the standard deviation would achieve such a goal". In particular, why it puts the transformed (standardized) data on the "same scale". The question hints at deeper issues (what else might have "worked", which is linked to what "worked" might even mean, mathematically?), but it seemed sensible to at least address the more straightforward aspects of why this procedure "works" - that is, achieves the claims made for it in the text.

The entry on row $i$ and column $j$ of a covariance matrix is the covariance between the $i^{th}$ and $j^{th}$ variables. Note that on a diagonal, row $i$ and column $i$, this becomes the covariance between the $i^{th}$ variable and itself - which is just the variance of the $i^{th}$ variable.

Let's call the $i^{th}$ variable $X_i$ and the $j^{th}$ variable $X_j$; I'll assume these are already centered so that they have mean zero. Recall that $$Cov(X_i, X_j) =\sigma_{X_i} \, \sigma_{X_j} \, Cor(X_i, X_j)$$

We can standardize the variables so that they have variance one, simply by dividing by their standard deviations. When standardizing we would generally subtract the mean first, but I already assumed they are centered so we can skip that step. Let $Z_i = \frac{X_i}{\sigma_{X_i}}$ and to see why the variance is one, note that

$$Var(Z_i) = Var\left(\frac{X_i}{\sigma_{X_i}}\right) = \frac{1}{\sigma_{X_i}^2}Var(X_i) = \frac{1}{\sigma_{X_i}^2} \sigma_{X_i}^2 = 1$$

Similarly for $Z_j$. If we take the entry in row $i$ and column $j$ of the covariance matrix for the standardized variables, note that since they are standardized:

$$Cov(Z_i, Z_j) =\sigma_{Z_i} \, \sigma_{Z_j} \, Cor(Z_i, Z_j) = Cor(Z_i, Z_j)$$

Moreover when we rescale variables in this way, addition (equivalently: subtraction) does not change the correlation, while multiplication (equivalently: division) will simply reverse the sign of the correlation if the factor (divisor) is negative. In other words correlation is unchanged by translations or scaling but is reversed by reflection. (Here's a derivation of those correlation properties, as part of an otherwise unrelated answer.) Since we divided by standard deviations, which are positive, we see that $Cor(Z_i, Z_j)$ must equal $Cor(X_i, X_j)$ i.e. the correlation between the original data.

Along the diagonal of the new covariance matrix, note that we get $Cov(Z_i, Z_i) = Var(Z_i) = 1$ so the entire diagonal is filled with ones, as we would expect. It's in this sense that the data are now "on the same scale" - their marginal distributions should look very similar, at least if they were roughly normally distributed to start with, with mean zero and with variance (and standard deviation) one. It is no longer the case that one variable's variability swamps the others. You could have divided by a different measure of spread, of course. The variance would have been a particularly bad choice due to dimensional inconsistency (think about what would have happened if you'd changed the units one of your variables was in, e.g. from metres to kilometres). Something like median absolute deviation (or an appropriate multiple of the MAD if you are trying to use it as a kind of robust estimator of the standard deviation) may have been more appropriate. But it still won't turn that diagonal into a diagonal of ones.

The upshot is that a method that works on the covariance matrix of standardized data, is essentially using the correlation matrix of the original data. For which you'd prefer to use on PCA, see PCA on correlation or covariance?

Answer 2

Why do we divide by the standard deviation
whats wrong with dividing by the variance?

as @Silverfish already pointed out in a comment, the standard deviation has the same unit as the measurements. Thus, dividing by standard deviation as opposed to variance, you end up with a plain number that tells you where your case is relative to average and spread as measured by mean and standard deviation. This is very close to the idea of $z$-values and the standard normal distribution: If the data are normally distributed, standardization will transform them to a standard normal distribution.

So: standardization (mean centering + scaling by standard deviation) makes sense if you consider the standard normal distribution sensible for your data.

Why not some other quantity? Like...the sum of absolute values? or some other norm...

Other quantities are used to scale data, but the procedure is called standardization only if it uses mean centering and dividing by standard deviation. Scaling is the generic term.

E.g. I work with spectroscopic data and know that my detector has a wavelength-dependent sensitivity and an (electronic) bias. Thus I calibrate by subtracting the offset (blank) signal and multiplying (dividing) by a calibration factor.

Also, I may be centering not to the mean but instead to some other baseline value, such as the mean of a control group instead of the grand mean. (Personally, I almost never standardize as my variates already have the same physical unit and are in the same order of magnitude)

See also: Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?

Answer 3

This link answers your question clearly, I guess: http://sebastianraschka.com/Articles/2014_about_feature_scaling.html

I quote a small piece:

Z-score standardization or Min-Max scaling?

“Standardization or Min-Max scaling?” - There is no obvious answer to this question: it really depends on the application.

For example, in clustering analyses, standardization may be especially crucial in order to compare similarities between features based on certain distance measures. Another prominent example is the Principal Component Analysis, where we usually prefer standardization over Min-Max scaling, since we are interested in the components that maximize the variance (depending on the question and if the PCA computes the components via the correlation matrix instead of the covariance matrix; but more about PCA in my previous article).

However, this doesn’t mean that Min-Max scaling is not useful at all! A popular application is image processing, where pixel intensities have to be normalized to fit within a certain range (i.e., 0 to 255 for the RGB color range). Also, typical neural network algorithm require data that on a 0-1 scale.