How does eigenvalues measure variance along the principal components in PCA? [duplicate]

Question

I understand that eigenvalues measure variance along the principal components.

Questions

1. How are eigenvalues and variance same for PCA?
2. What is the intuition behind this being the same?
3. What is the mathematical proof behind this theory?

Answer 1

We start from data covariance matrix $$ S = \mathbb E(XX^{T})- \mathbb E(X) \mathbb E(X)^{T}$$ Say $\mu$ is a column vector of the same dimension of $X$ and $\mu^{T}\mu = 1$, then $$\mu^{T}S\mu=\mu^{T}(\mathbb E(XX^{T})- \mathbb E(X) \mathbb E(X)^{T}) \mu = \mathbb E((\mu^{T} X)(\mu^{T} X)^{T})-\mathbb E(\mu^{T} X) \mathbb E(\mu^{T} X)^{T}$$ Note that $\mu^{T}X$ is the projection of $X$ onto $\mu$, such that

$$\mathbb E((\mu^{T} X)(\mu^{T} X)^{T})-\mathbb E(\mu^{T} X) \mathbb E(\mu^{T} X)^{T} = Var(\mu^{T} X)$$ Since $Var(\mu^{T} X)$ is simply a number, so we denote it as $\lambda$, so we have $$\mu^{T}S\mu = \lambda$$ Since $\mu^{T}\mu = 1$, so $$\mu \mu^{T} S\mu = S\mu= \lambda \mu$$ which means the $\mu$ that we define in the first place is actually an eigenvector of the data covariance matrix, and the eigenvalue of which is the variance that the data has in that direction.

I cannot think of an intuitive way to make sense of this, but once you are familiar with the math, I think you will accept this.

Answer 2

Variance:

Variance is the square of the deviation from zero, so the total variance of a vector is the sum of its squared values. https://en.wikipedia.org/wiki/Variance

Eigenvectors and Eigenvalues:

Eigenvectors are basis vectors that capture the inherent patterns that make up a dataset. By convention these are unit vectors, with norm = 1 so that they are the linear algebra equivalent of 1. Eigenvectors come in pairs, 'left' and 'right'. http://mathworld.wolfram.com/Eigenvalue.html

Convention (based on preferred orientation of data matrix as column matrix) is that the right eigenvector (I'll use the notation I'm familiar with $L^T$ which is used in various applied fields when talking about PCA) is the basis functions and is what is variously known as principal components, loadings, latent factors amongst many other. The right eigenvector can be projected onto any dataset with the same variables, so is useful for model building.

The left eigenvector ($S$) is the weights that each sample takes for each right eigenvector, and as such can be considered a series of functions for linear transformation of your original matrix. In practice this means that $S$ is the weightings that are given to each sample in order to construct the right eigenvector, which means that by definition they represent the total amplitude that each right eigenvector explains in that particular sample.

$$L^T=S^†D$$

Summing the individual variances explained per sample gives the total variance explained in the dataset.

Eigenvalues are a set of scalars such that a linear transformation S is scaled with the right eigenvectors produces the same result as multiplying the right eigenvectors themselves by that value

$$SL^T = \lambda L^T$$

Since linear algebra multiplication involves summation of the products of the row and column entries in the two multiplicands then multiplication by a scalar that is the total variance of the linear transform gives the same result. This means that eigenvalues are the variance of the by definition.

Eigendecomposition and PCA:

Thinking about it a bit more I realise that while NIPALS (see original answer below) is more intuitive for understanding how PCA is calculated, the SVD method is more intuitive for understanding eigenvalues themselves.

In SVD the data is decomposed into two sets of unit vector matrices with a diagnonal scaling matrix in between.

$$D = U*sv*L^T$$

the scores matrix $S$ in PCA is $U*sv$ and $L$ is the eigenvectors

The singular values are related to eigenvalues as:

$sv = \sqrt{ev}$

Note this relationship between the two is subject to constraints as described in https://math.stackexchange.com/questions/127500/what-is-the-difference-between-singular-value-and-eigenvalue

Original Answer using NIPALS:

For me the best algorithm for understanding PCA in an intuitive way is NIPALS

https://folk.uio.no/henninri/pca_module/pca_nipals.pdf

With the NIPALS approach the following steps are taken

Inner product of data $D$ to get covariance matrix (correlation in scaled appropriately) whose diagonal is the sum of squares $D^TD$

Project an initial vector of weights $W$ onto the data (various sources recommend using random samples to do this, I prefer the unit vector of the square root of the sum of squares). I will refer to the eigenvectors as $L$

$$L^T_0=W^†D$$ This gives an initial guess at the prinicpal component, which is then projected onto the data to reconstruct it based on the initial guess. $$D_{recon} = WL^T_0$$

The residual is calculated $D-D_{recon}$ and its sum of squares is calculated. OLS is then preformed until the sum of squares reaches a predefined stopping criteria, each time calculating the unit vector arising from projecting the updated weightings onto the data. This unit vector is the iterated principal component or eigenvector.

In NIPALS we subtract the final reconstructed data from the original data then use the residual to calculate PC2, and always proceed to the next PC with the final residual after iterating. This means all variance accounted for by $PC_i$ is removed from consideration by $PC_{>i}$

Initially the product of $WD$ is a vector with a norm that is not equal to 1, so we calculate the norm of the vector and use it to scale the vector to a unit vector.

The reason for creating unit vectors is that they are numericlly more stable than unconstrained vectors and have the nice property of behaving the same in linear algebra multiplication and inverse matrix operations (basically they are the linear algebra equivalent of the number 1).

How is eigenvalues and variance same for PCA?

So what is this norm that was used to scale the eignevector? It is the square root of the sum of squares of the coefficicents in the vector, i.e. the square root of the variance. The eigenvalue is the square of this value, i.e. it is the sum of squares = total variance.

If the characteristic vectors (the eigenvectors) are not unit vectors then the eigenvalues would not be their variance, but since we define eigenvectors as unit vectors then it falls out naturally that they are the variance of that vector in the data. If we calculate the scores by projecting the eigenvectors onto the data (note the formula below only works because L is comprised of unit vectors)

$$S = LD$$

Then the scores, since they have being multiplied by unit vectors, take on the total variance that is captured within the data by each unit vector. By definition eigenvalues are scalars that map the characteristic vectors of a matrix onto the matrix, i.e. they scale the vectors in order to reconstruct the original data based on the characteristic vectors.

Whats is the intuition behind this being the same.

Whats is the mathematical proof behind this theory?

They are this way by definition as described above