1
$\begingroup$

Long time reader, first time poster. Hopefully I won't screw this up...

In the context of Principal Component Analysis, I have the sense that the components of an eigenvector are a measure of the contribution of each variable to the variation on that axes. I just can't find anything that puts it in those terms. Would someone confirm and provide support, or provide a more accurate understanding?

[ Edit: adding for clarity... ]

In other words, given the following expression for the first principle component (hope I've gotten this right from memory) ...

$$Y_1 = e_{11} Z_1 + e_{12}Z_2 + \cdots + e_{1p}Z_p$$

Is it reasonable to interpret, for example, $e_{11}$ as a measure of the contribution of $Z_1$ to the variance of the data the first component?

Now that I've written it out in this form the answer seems obvious as $Y_i$ is just a linear combination of $e_{ij}$ and the standardized data...

$\endgroup$

2 Answers 2

2
$\begingroup$

Assume we are given $n$ random variables $X_1, ..., X_n$ that satisfy some conditions (so that the covariances exist and so forth) then to me, PCA means forming the matrix $C = (\text{Cov}(X_i, X_j))_{i,j=1,...,n}$. This is a symmetric, positive semidefinite matrix so it can be diagonalized using an orthogonal matrix. I.e. we can find an invertible matrix $A \in \mathbb{R}^{n \times n}$ such that $A^T = A^{-1}$ and $$A^{-1} C A = D = \text{diag}(\lambda_1, ..., \lambda_n)$$

One can show that, since $C$ is the covariance matrix of the vector of random variables $X = (X_1, ..., X_n)$ that then the covariance matrix of $AX$ is $A^TCA = D$. This is simple algebra using the mere definition and the bilinearity of covariance (maybe I got it wrong here and one needs to put $A^T$ in front of $X$ instead of $A$ or so). This means that we get new random variables $Y_1, ..., Y_n$ that are linear combinations of the old ones and such that $$\text{Cov}(Y_i, Y_j) = 0$$ for all $i \neq j$. However, for $i=j$, i.e. for the diagonal entries we have $$\lambda_i = D_{ii} = \text{Cov}(Y_i, Y_i) = \text{Var}(Y_i)$$

i.e. the $i$-th eigenvalue belonging to the $i$-th eigenvector is the variance of the $i$-th resulting PCA component (i.e. the new variable $Y_i$).

What people now do is take a threshold and throw away all $Y_i$ such that $\lambda_i$ is below that threshold. I.e. those variables who having a low variance 'do not contribute much'. However, this is just a heuristic (who sais that just because $Y_i$ has a small variance, it does not contribute much? It may be that this difference is small but unbelievably important ...) and should be used only when necessary, i.e. when you just have 30 or 40 variables and 1 mio. examples and you are using an algorithm that is able to detect uneccessary features then I would not use PCA as this tends to confuse the algorithm...

$\endgroup$
3
  • $\begingroup$ Also, there are many answers here that also connect this to the geometry, i.e. if the variables are multivariately Gaussian distributed then if you sample from them often enough you get a shape like an ellipse and the eigenvectors then "point to the longest sides" of that ellipse... Ill have to search for that thread... $\endgroup$ Commented Feb 6, 2019 at 16:16
  • $\begingroup$ This answer by amoeba explains it really well: stats.stackexchange.com/questions/2691/… $\endgroup$ Commented Feb 6, 2019 at 16:18
  • $\begingroup$ Thank you @Fabian. I agree this is a good summary of how PCA works. Amoeba's answer seems to be required reading on the subject! I think my question was poorly phrased and/or so trivial that it was misunderstood. Please have a look at the edits I made to the question. $\endgroup$ Commented Feb 7, 2019 at 0:40
0
$\begingroup$

Answer to the edited version of the question:

We must assume here that $Y_1$ is the original data and $Z_1, ..., Z_n$ are the PCA-variables. Then by the simple rule

$$\text{Var}(aX + bY) = a^2\text{Var}(X) + b^2\text{Var}(Y) + 2ab\text{Cov}(X,Y)$$

(Exercise: prove this rule ;-)) we have

$$\text{Var}(Y_1) = e_{11}^2\text{Var}(Z_1) + ... + e_{1p}^2\text{Var}(Z_p) + 2\sum_{i \neq j} e_{1i} e_{1j} \underbrace{\text{Cov}(Z_i, Z_j)}_{=0}$$

So all in all

$$\text{Var}(Y_1) = e_{11}^2\lambda_1 + ... + e_{1p}^2\lambda_p$$

i.e. the variance of the original data component is built up from the entries in the transformation matrix and all the eigenvalues...

Does that go into the right direction?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.