0
$\begingroup$

Why should the second PCA vector i.e. vector with largest variance in reduced subspace be orthogonal to the first PCA vector?

$\endgroup$

marked as duplicate by amoeba, mdewey, Peter Flom Nov 20 '16 at 14:22

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ How do you define "reduced subspace"? $\endgroup$ – Matthew Drury Nov 19 '16 at 16:32
  • 2
    $\begingroup$ How do you define "should"? $\endgroup$ – ttnphns Nov 19 '16 at 17:15
  • $\begingroup$ My point is that the phrase "reduced subspace" already presupposes some method for choosing one out of the infinitely many complementary subspaces to that spanned by the largest principal component. The most natural way to do so is with an inner product, and an orthogonal complement. $\endgroup$ – Matthew Drury Nov 19 '16 at 18:14
  • $\begingroup$ See also: stats.stackexchange.com/questions/130882. $\endgroup$ – amoeba Nov 20 '16 at 18:44
3
$\begingroup$

The first (second) PCA direction is given by the eigenvector of the covariance matrix of the data corresponding to the largest (second largest) eigenvalue, see also this question.

The covariance matrix is symmetric and eigenvectors of (real) symmetric matrices are orthogonal, which gives the result.

$\endgroup$
  • 2
    $\begingroup$ This is true, but I think of this as more consequence of the geometric definition than the definition of the principal components themselves. $\endgroup$ – Matthew Drury Nov 19 '16 at 16:33
6
$\begingroup$

From a simple geometric point of view: If the second eigenvector was not orthogonal to the first, then either the first eigenvector would not account for as much variation as possible, or the second eigenvector would not account for as much variation as possible.

Try drawing an ellipse and try drawing vectors through the ellipse where either the first or the second vector doesn't correspond to the principal axis of the ellipse.

Now try to imagine what the data points contained inside the ellipse would look like when projected unto those two vectors where one of the is slightly offset. You'll notice that in order to maximize variance along the direction of the vectors you've drawn, you HAVE to draw them orthogonally and you have to draw the first vector through the major axis of the ellipse.

$\endgroup$
  • 2
    $\begingroup$ This, in my opinion, is the more correct answer. $\endgroup$ – Matthew Drury Nov 19 '16 at 18:15
  • 1
    $\begingroup$ This does not make sense to me at all (cc to @MatthewDrury). In order for PC1 to account for maximum variance, it should be chosen along the major axis of the ellipse, here I agree. But then if you want to choose PC2 that accounts for maximum variance and do not require orthogonality you can certainly have more variance if you choose any other vector apart from the minor axis. $\endgroup$ – amoeba Nov 20 '16 at 8:10
  • 2
    $\begingroup$ @amoeba Fair point. I think the general idea is that you choose the second vector to account for the maximum amount of variance that has not yet been accounted for by the projection onto the first vector. I must admit, this does seem to beg the question. Possibly I'm horribly biased to prefer coordinate invariant, geometric definitions. Curse my mathematical upbringing. $\endgroup$ – Matthew Drury Nov 20 '16 at 23:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.