The idea of PCA is to find the directions (in high dimensional space) in which the essential structures (with regard to large variance, scatter) of the data lie. The assumption is that original features(variables) have a linear relationship. So, correlated original features(variables) are captured by PCA.

The steps of PCA are the following:

  1. Features are centred (but the direction does not change).
  2. The covariance matrix S (K x K) is calculated (it is symmetrical).
  3. The eigenvalue and eigenvector are calculated. Normalized eigenvectors correspond to loadings (weights).
  4. Principal components i.e. scores, are calculated. Scores are weighted sums of the observations on the original features. So represented by linear combination, where principal components (PC 1, PC 2 ... PC K) are orthogonal (because the covariance matrix is symmetrical).
  5. In the end you have to sort eigenvalues (variance) according to size and select principal components accordingly.

Now forget about everything I said before and assume that you've found PC 1(with regard to highest variance). As you know, the next principal component (i.e. PC 2) must be orthogonal to PC 1. So we automatically know the direction of PC 2, right? Since my space is K dimensional I will take the next principal component (PC 3) so that it is orthogonal to the first and second principal components, right? etc.

Now, could I say that if I had only determined the direction of the first principal component, all other direction of the principal components would be determined automatically?

  • 1
    $\begingroup$ It is a basic fact of linear algebra that In $n$ dimensions, the space of vectors orthogonal to a subspace of dimension $k$ has $n-k$ dimensions, not $1$ dimension (assuming the metric is nondegenerate, for otherwise there are even more orthogonal vectors). $\endgroup$ – whuber Sep 22 at 21:08
  • 1
    $\begingroup$ PC2 must be orthogonal to PC1. So we automatically know the direction of PC2, right? Not quite. You don't know the direction of PC2. You get to know the subspace orthogonal to PC1, the direction being somewhere within the subspace and has yet to be found. Be found by the same trick of the maximization that PC1 had been found out. $\endgroup$ – ttnphns Sep 22 at 23:52

Let's make a 10-dimensional example. Say that you got $\text{PC}_1=[1,1,1,1,1,1,1,1,1,1]$, what would $\text{PC}_2$ be?

Surely, it must be orthogonal to $\text{PC}_1$, but is that choice unique?

How can you differentiate between $\text{PC}_2^A=[-1,1,-1,1,-1,1,-1,1,-1,1]$ and $\text{PC}_2^B=[1,1,1,1,1,-1,-1,-1,-1,-1]$? Both are orthogonal to $\text{PC}_1$ after all.

See? The choice of $\text{PC}_2$ is not automatic given $\text{PC}_1$. In fact, it amounts to the same procedure to retrieve $\text{PC}_1$, after you remove the effect of $\text{PC}_1$ from data.

| cite | improve this answer | |

How about a more intuitive approach: You're thinking in 2D. The last component has no choice: it's determined by all of the other components. But intermediate components do have a choice. In the 2D case, the second choice is also the last choice, and therefore has no options: it must point in the direction that's determined by the prior choice.

Imagine a cloud of 3D points in space in front of you. Imagine your first component is a wooden rod that floats in space where you place it. You align this rod so that it lines up with the cloud's longest axis. Then you drill a hole in the side of this rod and place a second rod into it at a right angle. You can then rotate the original rod, pivoting the second rod through 360 degrees.

You can't make the second rod point in any direction, but you can pivot it through the circle perpendicular to the first rod. Your first rod constrains the second, but it does not absolutely determine where it points: you still have 360 degrees of rotation.

Now drill another hold at a right angle to the first two rods and insert your last rod. This one has no adjustability: your choices for the first two rods has totally constrained it instead of partially constrained it.

In higher dimensions, this is how it works. Your first component can point in any direction, your second component can point in any direction but it is constrained to a lower-dimensional range of choices than the first component. (Because you insist that the second must be perpendicular to the first.) The third is constrained by the first two -- it must be perpendicular to both, which cuts its choices down by two dimensions -- and so on.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.