2
$\begingroup$

This question already has an answer here:

I have a data matrix $X$ and I perform a PCA on this data with:

%// Subtract the mean from the data
Y = bsxfun(@minus, X, mean(X));

%// Obtain the PCA solution by calculate the SVD of Y
[U, S, V] = svd(Y);

%// Compute variance explained
rho = diag(S).^2./sum(diag(S).^2);

%// Plot variance explained
plot(rho, 'o-')
title('Variance explained by principal components');
xlabel('Principal component');
ylabel('Variance explained');

I get this plot

enter image description here

What is the reason for this to happen? It cannot be correct that only the first PC can explain all the variance.

My data looks like

price       crime      nox        rooms      dist          radial  proptax    stratio   lowstat   lprice     lnox      lproptax
==========================================================================================================================
24000       .006       5.38       6.57       4.09          1       29.6       15.3       4.98   10.08581   1.682688    5.69036  
21599       .027       4.69       6.42       4.97          2       24.2       17.8       9.14   9.980402   1.545433   5.488938  
34700       .027       4.69       7.18       4.97          2       24.2       17.8       4.03    10.4545   1.545433   5.488938  
33400       .032       4.58          7       6.06          3       22.2       18.7       2.94   10.41631   1.521699   5.402678  
...

so the values of first attribute are much higher than the rest. Can this be the reason to the weird "variance explained" plot?

|cite|improve this question
$\endgroup$

marked as duplicate by amoeba says Reinstate Monica, kjetil b halvorsen, Nick Cox, John, Silverfish Sep 29 '15 at 12:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 3
    $\begingroup$ Yes. The "total variance" of your dataset is simply the sum of the variances of all the variables. Your price variable will dominate the sum, and hence the PC1 will essentially equal to price and will explain almost all the variance. You probably want to run PCA on correlations not on covariances (see stats.stackexchange.com/questions/53), i.e. you probably should standardize your data after centering but prior to doing SVD: Y = bsxfun(@times, Y, 1./std(Y)); $\endgroup$ – amoeba says Reinstate Monica Sep 28 '15 at 16:52
  • 2
    $\begingroup$ No, that's just centering. The variances can still be very different, check for yourself by running var(Y). $\endgroup$ – amoeba says Reinstate Monica Sep 28 '15 at 16:55
  • 1
    $\begingroup$ PC1 is just a linear combination of your original variables. Of course you can see how they are related. If you don't standardize, it will be $\sim1$ times price plus $\sim 0$ times everything else. $\endgroup$ – amoeba says Reinstate Monica Sep 28 '15 at 17:00
  • 2
    $\begingroup$ See stats.stackexchange.com/questions/87037/… for another example in which uncritical use of covariance matrix produced garbage, which was not even noticed by all contributors to the thread. $\endgroup$ – Nick Cox Sep 28 '15 at 17:11
  • 1
    $\begingroup$ After some reflection, I edited the top answer in the "PCA on correlation or covariance" thread (to include some plots) and I think this question is now a duplicate of that one. In the dataset considered there PCA on covariance explains over 98% of the variance. $\endgroup$ – amoeba says Reinstate Monica Sep 29 '15 at 11:31

Browse other questions tagged or ask your own question.