Timeline for Why do we divide by the standard deviation and not some other standardizing factor before doing PCA?
Current License: CC BY-SA 3.0
21 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Jul 25, 2016 at 20:34 | comment | added | ttnphns | Suppose we aren't discussing to center or not to center (agreed to center), then why scale by dividing by a variable's s.d. and not other variable statistic (such as variance or mean abs deviation)? My answer is: Linear PCA is the least-squares diagonalizer tool for the X'X-type matrix. Then, if we want the matrix in its input view to have equal diagonal elements (we may or may not want, depends on our reasons) then division each variable by its s.d. (or value proportional to it) is the only way to achieve that. | |
| Jul 25, 2016 at 19:29 | answer | added | Esra | timeline score: 0 | |
| Feb 7, 2015 at 15:30 | answer | added | cbeleites | timeline score: 8 | |
| Jan 20, 2015 at 0:22 | comment | added | Silverfish | Re "why not divide by variance instead" - that can be fairly easily explained by the dimensional inconsistency. It would give you strange results if you changed the units one of the variables was in, for instance. Re "why not divide by MAD" - if the data were normally distributed, then since (in the population) MAD is proportional to SD, it would be possible to divide by an appropriate multiple of MAD and get an (inefficient but possibly robust?) estimate of the correlation. That's more interesting. | |
| Jan 20, 2015 at 0:11 | history | edited | Charlie Parker | CC BY-SA 3.0 |
added 592 characters in body
|
| Jan 20, 2015 at 0:05 | comment | added | amoeba | Funny, it does seem like an exact duplicate of ""Normalizing" variables for SVD / PCA" question, but unfortunately that one does not contain any answers discussing other possible normalization factors (apart from $\sigma$), so I am reluctant to mark this question as a duplicate of that one... On the other hand, for potential answerers it might be a good idea to provide a new answer in that thread and then mark this one as duplicate :) | |
| Jan 19, 2015 at 23:59 | comment | added | Charlie Parker | @amoeba thanks :) While I was thinking about this, since PCA can be derived from maximizing the variance, I guessed that dividing by a related quantity such as the STD, might be one of the reasons we divided by the STD. I then thought that maybe if we defined maybe a "variance" with any other norm, say L1, then we would divide by the STD of that norm. Though, it was just a guess and I am not 100% about this, hence the question. | |
| Jan 19, 2015 at 23:56 | answer | added | Silverfish | timeline score: 16 | |
| Jan 19, 2015 at 23:56 | history | edited | Charlie Parker | CC BY-SA 3.0 |
edited title
|
| Jan 19, 2015 at 23:55 | comment | added | amoeba | I should add that I think your question is very good (+1). One can indeed normalize by dividing by something else; for example, standard deviation is a very non-robust measure and can be misleading in the presence of strong outliers. So one can choose to divide by some robust measure of spread instead (see e.g. "median absolute deviation"). There is no "rigorous mathematical explanation" of why using STD is the best way to normalize, and you are right on the mark that it is "just an empirical observation" that it often works well. | |
| Jan 19, 2015 at 23:50 | comment | added | ttnphns | ... and thence MSCP (mean sum of squares & cross-products) matrix decomposed in PCA becomes the correlation matrix. | |
| Jan 19, 2015 at 23:48 | history | edited | Charlie Parker | CC BY-SA 3.0 |
added 96 characters in body
|
| Jan 19, 2015 at 23:46 | comment | added | ttnphns | The described form of standardization is called z-standardization (= converting to z-scores). The mean becomes 0 and st. deviation (and variance) becomes 1. | |
| Jan 19, 2015 at 23:44 | comment | added | amoeba | I did not want to edit myself, because I was not sure that I correctly understood your central question. Perhaps something like "Why normalizing the variables for PCA is done by dividing by standard deviation?" (or maybe "... by dividing specifically by standard deviation?"). | |
| Jan 19, 2015 at 23:44 | comment | added | ttnphns | stats.stackexchange.com/q/12200/3277 - one more link to add. | |
| Jan 19, 2015 at 23:38 | comment | added | Charlie Parker | @amoeba I would't mind changing the title. What do you suggest a better title would be? Something like: "Justifying why one divides by the standard deviation instead than by the variance or some other norm?" Btw, feel free to change the title if you find a better way to express it. Its sometimes hard to summarize everything I want to ask about in one sentence. | |
| Jan 19, 2015 at 23:37 | comment | added | amoeba | Regarding the terminology, "normalizing" is not a precise term and can refer to various things. Whereas "standardizing" means subtracting the mean and dividing by standard deviation, which is what you are referring to. | |
| Jan 19, 2015 at 23:35 | comment | added | amoeba | Your title question reads as if you are asking what the purpose of normalizing it (as opposed to not normalizing). This would be a duplicate of "PCA on correlation or covariance". However, what you actually seem to be asking is why normalization is done via dividing by STD (as opposed to dividing by variance, or range, etc.). If so, do you perhaps want to edit to make the title question more precise? | |
| Jan 19, 2015 at 23:33 | comment | added | Silverfish | There is a mathematical reason - dividing the (centered) data by the SD for each variable produces a transformed data set whose covariance matrix is simply the correlation matrix of the original (centered) data. After that, we're on correlation vs covariance matrix territory again. Are you seeking proof of how normalizing the data turns the covariance matrix into a correlation matrix? | |
| Jan 19, 2015 at 23:24 | history | asked | Charlie Parker | CC BY-SA 3.0 |