# Do components of PCA really represent percentage of variance? Can they sum to more than 100%?

When reading O'Reilly's Machine Learning For Hackers, it says each component represents a percentage of the variance. I've quoted the relevant part of the page below. Speaking to another expert they agreed it is the percentage.

However the 24 components sum to 133.2095%. How can that be?

(Excerpt from chapter 8, p.207)

Having convinced ourselves that we can use PCA, how do we do that in R? Again, this is a place where R shines: the entirety of PCA can be done in one line of code. We use the princomp function to run PCA:

pca <- princomp(date.stock.matrix[,2:ncol(date.stock.matrix)])


If we just type pca into R, we’ll see a quick summary of the principal components:

Call:
princomp(x = date.stock.matrix[, 2:ncol(date.stock.matrix)])
Standard deviations:
Comp.1 Comp.2 Comp.3 Comp.4 Comp.5 Comp.6 Comp.7
29.1001249 20.4403404 12.6726924 11.4636450 8.4963820 8.1969345 5.5438308
Comp.8 Comp.9 Comp.10 Comp.11 Comp.12 Comp.13 Comp.14
5.1300931 4.7786752 4.2575099 3.3050931 2.6197715 2.4986181 2.1746125
Comp.15 Comp.16 Comp.17 Comp.18 Comp.19 Comp.20 Comp.21
1.9469475 1.8706240 1.6984043 1.6344116 1.2327471 1.1280913 0.9877634
Comp.22 Comp.23 Comp.24
0.8583681 0.7390626 0.4347983
24 variables and 2366 observations.


In this summary, the standard deviations tell us how much of the variance in the data set is accounted for by the different principal components. The first component, called Comp.1, accounts for 29% of the variance, while the next component accounts for 20%. By the end, the last component, Comp.24, accounts for less than 1% of the variance. This suggests that we can learn a lot about our data by just looking at the first principal component.

Code and data can be found here:

https://github.com/johnmyleswhite/ML_for_Hackers/tree/master/08-PCA

-
I think the author's interpretation of the Standard deviations is slightly off. Since the standard deviations are in fact standard deviations, we must square them to see how much of the variance each component represents. The first component would represent $100\times\frac{29.1001249^2}{29.1001249^2+\cdots+0.4347983^2}$ percent of the total variance. –  Max Jul 24 '12 at 14:12
This question arises from two basic mistakes, I'm afraid: (1) it misses the heading announcing that the numbers are "Standard deviations" and mistakes them for variances and (2) it assumes those numbers are percents, but they are not. (Their units are whatever the stocks are measured in: dollars or percent change per year or whatever.) There's no bug here at all: the comment by @Max explains how to find percent of total variance. –  whuber Jul 24 '12 at 15:05
@whuber Perhaps I should have used "typo" instead of "bug"? :-) "Comp.1, accounts for 29% of the variance" is wrong and should read "Comp.1, accounts for 46% of the variance" –  Darren Cook Jul 24 '12 at 22:59
Thank you, Darren: I misunderstood that the confusion was present in the book and I took "bug" to refer to the R software itself. Finding that error was a good catch (I hope you found it rewarding to find out what's really going on with PCA)! –  whuber Jul 25 '12 at 15:16
Yes, this is unquestionably a bug in the book. There are a few places where I misused standard deviations instead of variances. (For example, there's a point in which we use RMSE instead of MSE to calculate R-squared.) I'm hoping we have time to sit down and correct these sorts of bugs in the near future. –  John Myles White Jul 26 '12 at 21:32
show 1 more comment

Use summary.princomp to see the "Proportion of Variance" and "Cumulative Proportion".

pca <- princomp(date.stock.matrix[,2:ncol(date.stock.matrix)])
summary(pca)

-
Thanks Joshua. So the first component is actually 46% of the variance. I'll send in a bug report to the book. –  Darren Cook Jul 24 '12 at 12:11
How is "Proportion of variance" calculated? The number shown is 0.4600083. But sqrt(pca$sdev[1]/sum(pca$sdev)) (roughly sqrt(29.1/133.2)) gives 0.4673904. –  Darren Cook Jul 24 '12 at 12:14
@DarrenCook: sdev implies that you're looking at the standard deviation, i.e. the square root of the variance (or $\sqrt{\lambda_i}$, using the notation from my answer), which should explain the difference. Try pca$sdev[1]^2/sum(pca$sdev^2) instead. –  MånsT Jul 24 '12 at 13:01
@DarrenCook: use the source... stats:::print.summary.princomp shows you that it squares the sdev component, which stats:::princomp.default shows is the sqrt of the eigen values. –  Joshua Ulrich Jul 24 '12 at 13:04
@MånsT Thanks, that was the magical formula! –  Darren Cook Jul 24 '12 at 13:12

They should sum to $100~\%.$

The total variance of a $p$-variate random variable $X$ with covariance matrix $\Sigma$ is defined as $${\rm tr}(\Sigma)=\sigma_{11}+\sigma_{22}+\cdots+\sigma_{pp}.$$

Now, the trace of a symmetric matrix is the sum of its eigenvalues $\lambda_1\geq\lambda_2\geq\ldots\geq\lambda_p.$ Thus the total variance is $${\rm tr}(\Sigma)=\lambda_1+\cdots+\lambda_p$$ if we use $\lambda_i$ to denote the eigenvalues of $\Sigma$. Note that $\lambda_p\geq 0$ since covariance matrices are positive-semidefinite, so that the total variance is non-negative.

But the principal components are given by $e_iX$, where $e_i$ is the $i$:th eigenvector (standardized to have length $1$), corresponding to the eigenvalue $\lambda_i$. Its variance is $${\rm Var}(e_iX)=e_i'\Sigma e_i=\lambda_ie_i'e_i=\lambda_i$$ and therefore the first $k$ principal components make up $$\Big(\frac{\lambda_1+\cdots\lambda_k}{\lambda_1+\cdots+\lambda_p}\cdot 100\Big)~\%$$ of the total variance. In particular, they make upp $100~\%$ of the total variance when $k=p$.

-
Did you see the (more recent) comment by @Max to the question? He nailed the answer. –  whuber Jul 24 '12 at 15:02
@whuber: I hadn't seen it, so thanks. I made a similar remark in a comment to Joshua's answer. –  MånsT Jul 24 '12 at 16:59

Here is some R code to complement previous answers (pca[["sdev"]] is usually written pca\$sdev, but it causes misformatting in the snippet below).

# Generate a dummy dataset.
set.seed(123)
x <- matrix(rnorm(400, sd=3), ncol=4)
# Note that princomp performs an unscaled PCA.
pca1 <- princomp(x)
# Show the fraction variance of each PC.
pca1[["sdev"]]^2
cumsum(pca1[["sdev"]]^2)/sum(pca1[["sdev"]]^2)
# Perform a scaled PCA.
pca2 <- princomp(x, cor=TRUE)
pca2[["sdev"]]^2
cumsum(pca2[["sdev"]]^2)/sum(pca2[["sdev"]]^2)


So, as @Max points out, working with the variance instead of the standard deviation and not forgetting to divide by the total variance solves the issue.

-