4
$\begingroup$

I have a data set ranged in different scales as well as some variables are sparse, for example,

n   V1  V2  V3      V4
0   0   1   34123   51523453
1   16  0   63124   34351234
2   0   0   63431   2343423
3   100 2   64351   34243
4   0   2   75283   35253523
5   0   1   2234    23423523
6   0   0   134523  315345
…   …   …   …       …

Because of the sparsity, I think I need to reduce the data dimension. Because of the different range, I would need to normalize the data.

To achieve these two goals, my original plan is to perform PCA whitening.

In the new decorrelated space, I would choose some eigenvectors associated with the first 2-3 largest eigenvalues as my principal vectors and reduce the dimension by projecting onto these vectors.

I think PCA whitening already normalizes the data in zero-mean and unit-variance manner.

I have two questions:

  1. Is it necessary to perform the normalization (e.g., subtract mean and divide by standard deviation independently) before performing the whitening?

  2. What other normalization techniques are worth to try?

Thanks in advance!!

Improve this question
$\endgroup$
4
  • $\begingroup$ For 1, I worry about losing any dependencies between variables if I take the normalization independently $\endgroup$
    – ben Heo
    Commented Oct 28, 2018 at 20:39
  • $\begingroup$ It's better to refer to this ("subtract mean and divide by standard deviation") as standardization. It's less ambiguous and has nice tie-ins with probability theory. $\endgroup$
    – shadowtalker
    Commented Oct 28, 2018 at 22:44
  • $\begingroup$ There is mix up in the question. First, usual PCA will not resque from "sparsity" because it centers the data columns therefore no sparsity will be preserved to struggle with. Second, PCA whitening is done to orthonormalize data and not to reduce dimensions. Yet, in any case you should scale your data simply because they are different units, not just because they are different variances. $\endgroup$
    – ttnphns
    Commented Oct 29, 2018 at 7:36
  • $\begingroup$ PCA whitening is performing PCA and computing standardazed pc scores. No other other PCA projection is needed after it. You must simply decide (i) do you want PCA to reduce dimensions and get uncorrelated data, (ii) if yes whether to normalize or not data initially (presumably yes) and whether you want the origin of pca rotation to be in the data centroid or in the zero value, (iii) whether after pca you will need unstandardized or standardized (normalized) pc scores. $\endgroup$
    – ttnphns
    Commented Oct 29, 2018 at 7:49

1 Answer 1

Reset to default
4
$\begingroup$

You should probably standardize your data before PCA.

PCA involves projecting the data onto the eigenvectors of the covariance matrix. If you don't standardize your data first, these eigenvectors will be all different lengths. Then the eigenspace of the covariance matrix will be "stretched", leading to similarly "stretched" projections. See here for an example of this effect. This is not what you want. See also here for several good answers describing the geometry of PCA.

However, there are situations in which you do want to preserve the original variances. See here for discussion on that topic.

As for your follow-up question, of whether you will lose dependencies between variables if you apply standardized independently: the answer is no. In fact, correlation between un-standardized random variables is equivalent to the covariance of standardized random variables.

Do note that covariance is inherently a measure of linear association. The covariance between a uniform random variable on $[-1, 1]$ and its square, for example, should be exactly 0. So higher-order relationships between variables could in fact be discarded by PCA. This is one motivation for kernel PCA.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Whether to standardize or not really depends on the situation (e.g. see this thread). For example, there are cases where one wouldn't want to standardize because preserving the variance of the original variables is important. $\endgroup$
    – user20160
    Commented Oct 28, 2018 at 23:40
  • 1
    $\begingroup$ @user20160 good point; updated. $\endgroup$
    – shadowtalker
    Commented Oct 28, 2018 at 23:55
  • $\begingroup$ Thanks a lot and providing kernal PCA link as well! Can you elaborate a little more on "correlation between un-standardized random variables is equivalent to the covariance of standardized random variables." or point me to? $\endgroup$
    – ben Heo
    Commented Oct 29, 2018 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.