1
$\begingroup$

After reviewing the many wonderful posts on Cross-Validated regarding understanding PCA, I was doing some of my own experiments, and I am a bit confused.

Consider a multivariate Gaussian in 3 dimensions, according to the Matlab code below:

P = 3;
Mu=zeros(1,P);
Sigma = eye(P)*500;
gm0 = gmdistribution(Mu,Sigma);


N=10E3;
X=gm0.random(N);

PCA is a linear transformation of the data, so that the resulting principal components are 1) uncorrelated and 2) the largest variance is in the first principal component, and then in decreasing order for the rest of the PCs. I also read about it being a "diagonalization" of the covariance matrix.

Here, the sample covariance matrix is (almost) diagonal, as it was generated from a diagonal covariance matrix Gaussian:

>> cov(X)

ans =

  492.0967    4.0832    2.5630
    4.0832  496.2937   -1.0372
    2.5630   -1.0372  496.0279

It seems like each column of X is essentially already a principal component. After all, they are practically uncorrelated. Indeed, the resulting projection looks just as Gaussian as the original data. But when I look at the loadings:

Xc = bsxfun(@minus,X,mean(X)); %It's already mean centered, but just in case
[U S V ] = svd(Xc);
US = U*S;
V
V =

    0.5615   -0.1746   -0.8088
    0.7994    0.3667    0.4758
    0.2135   -0.9138    0.3455

I find it hard to make sense of these eigenvectors. Indeed, the loadings do not seem to map nicely to each of the dimensions of the multivariate Gaussian. I thought it would essentially be an identity matrix.

Could you help me understand why? I imagine I'm missing something quite simple.

Perhaps more generally, how can I make data that is exactly the same as its principal compoenents? This was my attempt, and clearly it did not succeed--each of the original dimensions are getting mixed together.

$\endgroup$
  • 1
    $\begingroup$ You should check out this post on "whitening", which may help. (Also, note that your finite sample is not already "mean centered" ... and similarly the sample covariance matrix is not the identity either.) $\endgroup$ – GeoMatt22 Oct 8 '16 at 2:17
  • $\begingroup$ Thanks for the link, @GeoMatt22. Whitening is something I have done before but didn't know that's what it is called, so that is very helpful. However, my question still stands. Can you give me any insight? $\endgroup$ – user310374 Oct 11 '16 at 2:18
  • $\begingroup$ I will try to find time to answer as well, so stay tuned :) Am very busy these days. $\endgroup$ – amoeba Oct 11 '16 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.