Let $X$ be a data matrix of size $n \times p$.

Assume that $X$ is centered (column means subtracted). Then, the $p \times p$ covariance matrix is given by $$C = \frac{X^TX}{n-1}$$

Since $C$ is symmetric, it is diagonalizable, hence, $\exists$ a matrix $V$ such that $$C = VLV^T$$where $V$ is a matrix of eigenvectors of C and $L$ is diagonal with eigenvalues $\lambda_i$ of $C$.

Now, I've read things along the lines of: eigenvalues $λ_i$ show variances of the respective PCs. What does this mean? How is the spread/variance of a covariate related to the eigenvalue?

I understand that we want components with a large variance as large variance means more information (generally), but how does this relate to eigenvalues?


1 Answer 1


The variance of any $p$-vector $x$ is given by

$$\operatorname{Var}(x) = x^\prime C x.\tag{1}$$

We may write $x^\prime$ as a linear combination of the rows of $V,$ $v_1,$ $v_2,\ldots,$ $v_p,$ because

$$x^\prime = x^\prime\mathbb{I} = x^\prime V V^\prime = (x^\prime V)_1v_1 + (x^\prime V)_2v_2 + \cdots + (x^\prime V)_pv_p.$$

The coefficient of $v_i$ in this linear combination is $(x^\prime V)_i = (V^\prime x)_i.$

The diagonalization permits you to rewrite these relations more simply as

$$\operatorname{Var}(x) = x^\prime(V\Lambda V^\prime) x = \sum_{i=1}^p \lambda_{ii} (V^\prime x)_i^2.$$

In other words, the variance of $x$ is found as the sum of $p$ terms, each obtained by

(a) transforming to $y=V^\prime x,$ then (b) squaring each coefficient $y_i,$ and (c) multiplying the square by $\lambda_{ii}$.

This enables us to understand the action of $C$ in simple terms: $y$ is just another way of expressing $x$ (it uses the row vectors of $V$ as a basis) and its terms contribute their squares to the variance, weighted by $\lambda_{ii}.$

The relationship to PCA is the following. It makes little sense to maximize the variance, because by scaling $x$ we can make the variance arbitrarily large. But if we think of $x$ solely as determining a linear subspace, (if you like, an unsigned direction) we may represent that direction by scaling $x$ to have unit length. Thus, assume $||x||^2=1.$ Because $V$ is an orthogonal matrix, $y$ also has unit length:

$$||y||^2 = y^\prime y = (V^\prime x)^\prime(V^\prime x) = x^\prime(VV^\prime) x = x^\prime \mathbb{I}x = ||x||^2= 1.$$

To make the variance of $x$ as large as possible, you want to put as much weight as possible on the largest eigenvalue (the largest $\lambda_{ii}$). Without any loss of generality you can arrange the rows of $V$ so that this is $\lambda_{11}.$ A variance-maximizing vector therefore is $y^{(1)} = (1,0,\ldots,0)^\prime.$ The corresponding $x$ is

$$x^{(1)} = V y^{(1)},$$

the first column of $V.$ This is the first principal component. Its variance is $\lambda_{11}.$ By construction, it is a unit vector with the largest possible variance. It represents a linear subspace.

The rest of the principal components are obtained similarly from the other columns of $V$ because (by definition) those columns are mutually orthogonal.

When all the $\lambda_{ii}$ are distinct, this method gives a unique set of solutions:

The principal components of $C$ are the linear subspaces corresponding to the columns of $V.$ The variance of column $i$ is $\lambda_{ii}.$

More generally, there may be infinitely many ways to diagonalize $C$ (this is when there are one or more eigenspaces of dimension greater than $1,$ so-called "degenerate" eigenspaces). The columns of any particular such $V$ still enjoy the foregoing properties. $V$ is usually chosen so that $\lambda_{11}\ge\lambda_{22}\ge\cdots\ge\lambda_{pp}$ are the principal components in order.

  • $\begingroup$ Thanks for your answer. What exactly is the $p-$vector $x$ here (with respect to my data matrix)? Why is the variance defined like so? $\endgroup$
    – the man
    Jul 29, 2020 at 7:58
  • $\begingroup$ $x$ is literally any vector. That this is the formula for the variance is a consequence of its bilinearity properties. See stats.stackexchange.com/a/185000/919 or (for a more general and abstract explanation) math.stackexchange.com/a/3392345/1489. $\endgroup$
    – whuber
    Jul 29, 2020 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.