Do components of PCA really represent percentage of variance? Can they sum to more than 100%?

Question

When reading O'Reilly's Machine Learning For Hackers, it says each component represents a percentage of the variance. I've quoted the relevant part of the page below. Speaking to another expert they agreed it is the percentage.

However the 24 components sum to 133.2095%. How can that be?

---

(Excerpt from chapter 8, p.207)

Having convinced ourselves that we can use PCA, how do we do that in R? Again, this is a place where R shines: the entirety of PCA can be done in one line of code. We use the princomp function to run PCA:

pca <- princomp(date.stock.matrix[,2:ncol(date.stock.matrix)])

If we just type pca into R, we’ll see a quick summary of the principal components:

Call:
princomp(x = date.stock.matrix[, 2:ncol(date.stock.matrix)])
Standard deviations:
Comp.1 Comp.2 Comp.3 Comp.4 Comp.5 Comp.6 Comp.7
29.1001249 20.4403404 12.6726924 11.4636450 8.4963820 8.1969345 5.5438308
Comp.8 Comp.9 Comp.10 Comp.11 Comp.12 Comp.13 Comp.14
5.1300931 4.7786752 4.2575099 3.3050931 2.6197715 2.4986181 2.1746125
Comp.15 Comp.16 Comp.17 Comp.18 Comp.19 Comp.20 Comp.21
1.9469475 1.8706240 1.6984043 1.6344116 1.2327471 1.1280913 0.9877634
Comp.22 Comp.23 Comp.24
0.8583681 0.7390626 0.4347983
24 variables and 2366 observations.

In this summary, the standard deviations tell us how much of the variance in the data set is accounted for by the different principal components. The first component, called Comp.1, accounts for 29% of the variance, while the next component accounts for 20%. By the end, the last component, Comp.24, accounts for less than 1% of the variance. This suggests that we can learn a lot about our data by just looking at the first principal component.

---

Code and data can be found here:

https://github.com/johnmyleswhite/ML_for_Hackers/tree/master/08-PCA

Answer 1

Use summary.princomp to see the "Proportion of Variance" and "Cumulative Proportion".

pca <- princomp(date.stock.matrix[,2:ncol(date.stock.matrix)])
summary(pca)

Answer 2

They should sum to $100~\%.$

The total variance of a $p$-variate random variable $X$ with covariance matrix $\Sigma$ is defined as $${\rm tr}(\Sigma)=\sigma_{11}+\sigma_{22}+\cdots+\sigma_{pp}.$$

Now, the trace of a symmetric matrix is the sum of its eigenvalues $\lambda_1\geq\lambda_2\geq\ldots\geq\lambda_p.$ Thus the total variance is $${\rm tr}(\Sigma)=\lambda_1+\cdots+\lambda_p$$ if we use $\lambda_i$ to denote the eigenvalues of $\Sigma$. Note that $\lambda_p\geq 0$ since covariance matrices are positive-semidefinite, so that the total variance is non-negative.

But the principal components are given by $e_iX$, where $e_i$ is the $i$:th eigenvector (standardized to have length $1$), corresponding to the eigenvalue $\lambda_i$. Its variance is $${\rm Var}(e_iX)=e_i'\Sigma e_i=\lambda_ie_i'e_i=\lambda_i$$ and therefore the first $k$ principal components make up $$\Big(\frac{\lambda_1+\cdots\lambda_k}{\lambda_1+\cdots+\lambda_p}\cdot 100\Big)~\%$$ of the total variance. In particular, they make upp $100~\%$ of the total variance when $k=p$.

Answer 3

Here is some R code to complement previous answers (pca[["sdev"]] is usually written pca$sdev, but it causes misformatting in the snippet below).

# Generate a dummy dataset.
set.seed(123)
x <- matrix(rnorm(400, sd=3), ncol=4)
# Note that princomp performs an unscaled PCA.
pca1 <- princomp(x)
# Show the fraction variance of each PC.
pca1[["sdev"]]^2
cumsum(pca1[["sdev"]]^2)/sum(pca1[["sdev"]]^2)
# Perform a scaled PCA.
pca2 <- princomp(x, cor=TRUE)
pca2[["sdev"]]^2
cumsum(pca2[["sdev"]]^2)/sum(pca2[["sdev"]]^2)

So, as @Max points out, working with the variance instead of the standard deviation and not forgetting to divide by the total variance solves the issue.