2
$\begingroup$

Normally I expect most of the variance or at least a quarter of the variance to be explained in the first 2 dimensions of the PCA. However, recently I came across a PCA where the first was only about 12% and the 2 about 6%. What does this mean exactly? Does this mean that the PCA in question is not good enough to explain most of the variance?

$\endgroup$
3
  • 3
    $\begingroup$ How many variables were there? If you have enough variables, 12% might be impressive. $\endgroup$ May 3 '20 at 8:24
  • $\begingroup$ @gung-ReinstateMonica hmm I never thought about that. What would be consider many? I have a total of ~1500 points. $\endgroup$
    – Ahdee
    May 3 '20 at 15:10
  • 1
    $\begingroup$ I don't mean the number of points, I mean the number of variables. 1500 points is fine. $\endgroup$ May 3 '20 at 15:27
5
$\begingroup$

The PCA is not good or bad. The values you have, where two principal components do not explain a big part of variance, mean that the data is far from being near a 2 dimensional subspace.

I can understand your data have many dimensions, as the variance of the second component only explains 6% and rest of components must explain even less each.

Here is an example. I create 1000 points in 5 dimensions with center (0,0,0,0,0) and different covariance matrices. In the first case, they are all distributed equally along each of the 5 dimensions. In the second case, points are also in 5 dimensions, but mostly near a plane in 2d. You can see the explained variance of each component.

data_5d = mvrnorm(1000, c(0,0,0,0,0), diag(5))
summary(prcomp(data_5d))
# Importance of components:
#                          PC1    PC2    PC3    PC4    PC5
# Standard deviation     1.027 1.0064 0.9928 0.9604 0.9511
# Proportion of Variance 0.216 0.2076 0.2020 0.1890 0.1854
# Cumulative Proportion  0.216 0.4236 0.6256 0.8146 1.0000

data_almost_2d = mvrnorm(1000, c(0,0,0,0,0), diag(c(1,1,0.1,0.1,0.1)))
summary(prcomp(data_almost_2d))
# Importance of components:
#                           PC1    PC2     PC3     PC4     PC5
# Standard deviation     1.0074 0.9724 0.32140 0.30981 0.30016
# Proportion of Variance 0.4511 0.4203 0.04591 0.04266 0.04005
# Cumulative Proportion  0.4511 0.8714 0.91729 0.95995 1.00000
$\endgroup$
5
  • 1
    $\begingroup$ Agreed: It's all about what is and what is not there in the data. I'd argue that the most successful applications of PCA are with a bundle of variables known to be measures of, if not the broadly the same thing, then of two or three similar things. Very often PCA is a kind of fishing expedition in which people throw in all the data to hand and hope that PCA will discover structure they didn't know about. Also, as another answer makes plain. nonlinearities won't help. $\endgroup$
    – Nick Cox
    May 3 '20 at 8:22
  • 1
    $\begingroup$ Let's focus on the case of PCA from a correlation matrix or equivalently on variables each standardized first so that each variance is 1. Then if all the variables are pretty much uncorrelated with each other, then each PC will also have a variance near 1. But if there are strong correlations the highest PCs will have variances much bigger than 1 and the lowest PCs may have variances much closer to 0. So, it's not so much about whether the lowest PCs have low variance -- that could be a good thing! -- but about the variation of the variances. For variance, here read eigenvalue too. $\endgroup$
    – Nick Cox
    May 3 '20 at 9:20
  • $\begingroup$ Thanks for the post javierazcoiti and thank @NickCox - so if understand what you said, having a lot of uncorrelated values could result in lower variance in the lower PCs? $\endgroup$
    – Ahdee
    May 3 '20 at 15:19
  • $\begingroup$ Not what I said. Uncorrelated variables (not values) means all PCs have the same variance. Samplng variation will mean that eigenvalues will differ somewhat. $\endgroup$
    – Nick Cox
    May 3 '20 at 15:26
  • $\begingroup$ @NickCox ok got that makes much more sense. Thank you this is helpful. $\endgroup$
    – Ahdee
    May 3 '20 at 17:04
1
$\begingroup$

If the goal is dimensionality reduction, PCA may not be a good option in this case according to the variances you found. Some non-linear coordinate transformation may be much more effective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.