Timeline for Why do we divide by the standard deviation and not some other standardizing factor before doing PCA?
Current License: CC BY-SA 3.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Jan 20, 2015 at 14:21 | comment | added | Silverfish | @amoeba That's quite correct; I happened to pick an unfortunate choice for the covariance matrix in the simulation matrix which obscured that. I rolled back to avoid confusion. | |
| Jan 20, 2015 at 14:02 | history | edited | Silverfish | CC BY-SA 3.0 |
remove typo
|
| Jan 20, 2015 at 14:02 | history | rollback | Silverfish |
Rollback to Revision 4
|
|
| Jan 20, 2015 at 12:51 | comment | added | amoeba | I am confused by your latest update. Are you saying that if the data cloud is rotated then the eigenvalues of the correlation matrix will always stay the same? I am either missing something, or this cannot be true. Imagine a 2D Gaussian data cloud strongly elongated along the x-axis. Standardizing both x and y will make it spherical, i.e. with eigenvalues 1 and 1. But if we first rotate it 45 degrees and then standardize, it will remain strongly stretched along the diagonal, with eigenvalues close to 2 and 0. Not sure how does fits to your simulation (which I only briefly looked at). | |
| Jan 20, 2015 at 3:01 | history | edited | Silverfish | CC BY-SA 3.0 |
remove typo
|
| Jan 20, 2015 at 2:25 | history | edited | Silverfish | CC BY-SA 3.0 |
note problem with rotation
|
| Jan 20, 2015 at 1:06 | history | edited | Silverfish | CC BY-SA 3.0 |
hedge intro a bit more
|
| Jan 20, 2015 at 0:59 | comment | added | ttnphns | @amoeba, Linear PCA needs the matrix to be the SSCP-type matrix. Any linear transform of the original variables preserves this type. Of course, you could do any nonlinear transform as well (such as, for instance, ranking, to get Spearman rho matrix), but then component scores and loadings loose their direct (in sense of least squares minimization) connection with the data: they now represent the transformed data instead! | |
| Jan 20, 2015 at 0:56 | history | edited | Silverfish | CC BY-SA 3.0 |
appropriate link plus hedge intro a bit
|
| Jan 20, 2015 at 0:43 | comment | added | amoeba | @ttnphns: If different variables have completely incomparable scales (temperature, length, weight, etc.), then the desire to somehow normalize the variables is quite understandable. That's the common argument for using correlation matrix instead of covariance matrix. But if somebody is worried about outliers, I see nothing wrong with subtracting the median instead of the mean and dividing by MAD instead of SVD... I never did it myself, but I think it does sound like a reasonable thing to do. | |
| Jan 20, 2015 at 0:34 | history | edited | Silverfish | CC BY-SA 3.0 |
hedge about "looking same"
|
| Jan 20, 2015 at 0:30 | comment | added | ttnphns | @amoeba, why not divide by MAD or by variance instead of SD is, essentially, the same question as why to differentially scale at all: that is, why not to do PCA on covariances instead? I support this idea in the preceding comment. | |
| Jan 20, 2015 at 0:11 | comment | added | Silverfish | @ameoba On the "medium deep" point, the fact that we get variances of one down the diagonal of the new covariance matrix is essentially what we mean by getting the transformed data to have variables "on the same scale" from the PCA perspective. On the "very deep" issues raised by this question, I'm not sure there is much difference between asking "well why do we use variances as our measure of scale in PCA?" and asking "why does PCA concern itself with (co)variances?" - or at least, that the two issues would be intimately related. | |
| Jan 20, 2015 at 0:03 | comment | added | Silverfish | @ameoba That is correct. At present I think the question is not quite clear, this is only an answer on the narrow point: the OP asks "how does dividing by the standard deviation achieve that" (the "that" refers to the text, which mentions things like how does the variance become unit). | |
| Jan 19, 2015 at 23:59 | comment | added | amoeba | I think this answer does not really touch on the actual (and non-trivial) question of why standard deviation is taken as a measure of spread and used for normalization. Why not taking median absolute deviation instead? Granted, the resulting covariance matrix will not be the "default" correlation matrix, but perhaps it will be better, e.g. a more robust estimation of the correlation matrix. See also my last comment to the OP. | |
| Jan 19, 2015 at 23:56 | history | answered | Silverfish | CC BY-SA 3.0 |