Normally I expect most of the variance or at least a quarter of the variance to be explained in the first 2 dimensions of the PCA. However, recently I came across a PCA where the first was only about 12% and the 2 about 6%. What does this mean exactly? Does this mean that the PCA in question is not good enough to explain most of the variance?
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ How many variables were there? If you have enough variables, 12% might be impressive. $\endgroup$– gung - Reinstate MonicaCommented May 3, 2020 at 8:24
-
$\begingroup$ @gung-ReinstateMonica hmm I never thought about that. What would be consider many? I have a total of ~1500 points. $\endgroup$– AhdeeCommented May 3, 2020 at 15:10
-
1$\begingroup$ I don't mean the number of points, I mean the number of variables. 1500 points is fine. $\endgroup$– gung - Reinstate MonicaCommented May 3, 2020 at 15:27
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
The PCA is not good or bad. The values you have, where two principal components do not explain a big part of variance, mean that the data is far from being near a 2 dimensional subspace.
I can understand your data have many dimensions, as the variance of the second component only explains 6% and rest of components must explain even less each.
Here is an example. I create 1000 points in 5 dimensions with center (0,0,0,0,0) and different covariance matrices. In the first case, they are all distributed equally along each of the 5 dimensions. In the second case, points are also in 5 dimensions, but mostly near a plane in 2d. You can see the explained variance of each component.
data_5d = mvrnorm(1000, c(0,0,0,0,0), diag(5))
summary(prcomp(data_5d))
# Importance of components:
# PC1 PC2 PC3 PC4 PC5
# Standard deviation 1.027 1.0064 0.9928 0.9604 0.9511
# Proportion of Variance 0.216 0.2076 0.2020 0.1890 0.1854
# Cumulative Proportion 0.216 0.4236 0.6256 0.8146 1.0000
data_almost_2d = mvrnorm(1000, c(0,0,0,0,0), diag(c(1,1,0.1,0.1,0.1)))
summary(prcomp(data_almost_2d))
# Importance of components:
# PC1 PC2 PC3 PC4 PC5
# Standard deviation 1.0074 0.9724 0.32140 0.30981 0.30016
# Proportion of Variance 0.4511 0.4203 0.04591 0.04266 0.04005
# Cumulative Proportion 0.4511 0.8714 0.91729 0.95995 1.00000
-
1$\begingroup$ Agreed: It's all about what is and what is not there in the data. I'd argue that the most successful applications of PCA are with a bundle of variables known to be measures of, if not the broadly the same thing, then of two or three similar things. Very often PCA is a kind of fishing expedition in which people throw in all the data to hand and hope that PCA will discover structure they didn't know about. Also, as another answer makes plain. nonlinearities won't help. $\endgroup$– Nick CoxCommented May 3, 2020 at 8:22
-
1$\begingroup$ Let's focus on the case of PCA from a correlation matrix or equivalently on variables each standardized first so that each variance is 1. Then if all the variables are pretty much uncorrelated with each other, then each PC will also have a variance near 1. But if there are strong correlations the highest PCs will have variances much bigger than 1 and the lowest PCs may have variances much closer to 0. So, it's not so much about whether the lowest PCs have low variance -- that could be a good thing! -- but about the variation of the variances. For variance, here read eigenvalue too. $\endgroup$– Nick CoxCommented May 3, 2020 at 9:20
-
$\begingroup$ Thanks for the post javierazcoiti and thank @NickCox - so if understand what you said, having a lot of uncorrelated values could result in lower variance in the lower PCs? $\endgroup$– AhdeeCommented May 3, 2020 at 15:19
-
$\begingroup$ Not what I said. Uncorrelated variables (not values) means all PCs have the same variance. Samplng variation will mean that eigenvalues will differ somewhat. $\endgroup$– Nick CoxCommented May 3, 2020 at 15:26
-
$\begingroup$ @NickCox ok got that makes much more sense. Thank you this is helpful. $\endgroup$– AhdeeCommented May 3, 2020 at 17:04
$\begingroup$
$\endgroup$
If the goal is dimensionality reduction, PCA may not be a good option in this case according to the variances you found. Some non-linear coordinate transformation may be much more effective.
