After reviewing the many wonderful posts on Cross-Validated regarding understanding PCA, I was doing some of my own experiments, and I am a bit confused.
Consider a multivariate Gaussian in 3 dimensions, according to the Matlab code below:
P = 3; Mu=zeros(1,P); Sigma = eye(P)*500; gm0 = gmdistribution(Mu,Sigma); N=10E3; X=gm0.random(N);
PCA is a linear transformation of the data, so that the resulting principal components are 1) uncorrelated and 2) the largest variance is in the first principal component, and then in decreasing order for the rest of the PCs. I also read about it being a "diagonalization" of the covariance matrix.
Here, the sample covariance matrix is (almost) diagonal, as it was generated from a diagonal covariance matrix Gaussian:
>> cov(X) ans = 492.0967 4.0832 2.5630 4.0832 496.2937 -1.0372 2.5630 -1.0372 496.0279
It seems like each column of
X is essentially already a principal component. After all, they are practically uncorrelated. Indeed, the resulting projection looks just as Gaussian as the original data. But when I look at the loadings:
Xc = bsxfun(@minus,X,mean(X)); %It's already mean centered, but just in case [U S V ] = svd(Xc); US = U*S; V V = 0.5615 -0.1746 -0.8088 0.7994 0.3667 0.4758 0.2135 -0.9138 0.3455
I find it hard to make sense of these eigenvectors. Indeed, the loadings do not seem to map nicely to each of the dimensions of the multivariate Gaussian. I thought it would essentially be an identity matrix.
Could you help me understand why? I imagine I'm missing something quite simple.
Perhaps more generally, how can I make data that is exactly the same as its principal compoenents? This was my attempt, and clearly it did not succeed--each of the original dimensions are getting mixed together.