I understand the PCA method in a following way:

0) I have a cloud of dots in ND space

1) I should find a vector which gives me a maximal variance when dots are projected on this vector.

2) I should find the second vector orthogonal to the first, which will explain the maximal possible variance.

And so on..

My question is - if I would use the plane, defined by first and second principal components, will it be the plane, which explains the maximum possible variance?


marked as duplicate by kjetil b halvorsen, whuber Jun 11 '18 at 13:53

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ yes, by definition because it is least squares minimisation of the sum of squares that is used to guide the calculation of the PCA. NIPALS is probably the most intuitive algorithm for understanding this as it rigidly calculates PCs in order while some other algorithms seek to optimise globally and are then ranked in order of variance explained. Either way the final PCs are explicitly ranked by variance explained, so the 1st and 2nd will explain the most variance that any 2 pcs possibly can. $\endgroup$ – ReneBt Jun 11 '18 at 8:37
  • $\begingroup$ That is my problem. I understand, that PC1,2 explain the highest possible amount of variance sequentially, but I could not understand why it is not possible that some vectors V1, which explains less variance than PC1, and V2 which explains more variance, that PC2, could not explain more variance TOGETHER than PC1 and PC2 TOGETHER. $\endgroup$ – zlon Jun 11 '18 at 8:45
  • $\begingroup$ It is perfectly possible that creating a hyperplane from multiple PCs together could. If the following sequence holds % Var explained: PC1:5 = 20, 18, 17, 16, 15 then PC1 and 2 will explain 38%, 3-5 will explain 48%. If you add any arbitrary amount of small numbers together you can get a bigger sum than specifically adding 2 largish number together. As long as PC1 and PC2 contribute <50% of variance explained then the rest of the PCs combined will explain more. $\endgroup$ – ReneBt Jun 11 '18 at 8:51
  • $\begingroup$ the question was is it true that plane defined by PC1 and PC2 explains the highest amount of variance, than any other possible plane. $\endgroup$ – zlon Jun 11 '18 at 8:52
  • 1
    $\begingroup$ seems this question has been asked before: stats.stackexchange.com/questions/347646/… $\endgroup$ – ReneBt Jun 11 '18 at 9:09

The answer is yes.

The most important ingredient of PCA analysis is the data covariance matrix

$$S = \mathbb E(XX^{T})-\mathbb E(X) (\mathbb E(X))^{T} $$

Then by choosing a vector $\mu$ in the same space of $X$, we can calculate

$$\mu^{T} S \mu = \mathbb E((\mu^{T}) X(\mu^{T} X)^{T})-\mathbb E(\mu^{T} X) (\mathbb E(\mu^{T} X))^{T}$$

Since $ \mu X$ is a number so that $\mu^{T} S \mu$ is the variance of "the cloud of data " has in the direction $\mu$.

Note also, $S$ is positive semi-definite, so that it can always be diagonalized

$$S = P^{T}\Lambda P$$

in which $P$ is an orthogonal matrix, and $\Lambda$ is a diagonal matrix whose diagonal entries are the variances.

so we have

$$\mu^{T} S \mu = (P\mu)^{T} \Lambda (P\mu)$$

it can be easily seen from $(P\mu)^{T} \Lambda (P\mu)$, if we project the data onto a plane, the maximum variance will be captured when the plane is spanned by the two leading principal components.


Not the answer you're looking for? Browse other questions tagged or ask your own question.