Understanding the determination of principal components

Question

The idea of PCA is to find the directions (in high dimensional space) in which the essential structures (with regard to large variance, scatter) of the data lie. The assumption is that original features(variables) have a linear relationship. So, correlated original features(variables) are captured by PCA.

The steps of PCA are the following:

1. Features are centred (but the direction does not change).
2. The covariance matrix S (K x K) is calculated (it is symmetrical).
3. The eigenvalue and eigenvector are calculated. Normalized eigenvectors correspond to loadings (weights).
4. Principal components i.e. scores, are calculated. Scores are weighted sums of the observations on the original features. So represented by linear combination, where principal components (PC 1, PC 2 ... PC K) are orthogonal (because the covariance matrix is symmetrical).
5. In the end you have to sort eigenvalues (variance) according to size and select principal components accordingly.

Now forget about everything I said before and assume that you've found PC 1(with regard to highest variance). As you know, the next principal component (i.e. PC 2) must be orthogonal to PC 1. So we automatically know the direction of PC 2, right? Since my space is K dimensional I will take the next principal component (PC 3) so that it is orthogonal to the first and second principal components, right? etc.

Now, could I say that if I had only determined the direction of the first principal component, all other direction of the principal components would be determined automatically?

Answer 1

Let's make a 10-dimensional example. Say that you got $\text{PC}_1=[1,1,1,1,1,1,1,1,1,1]$, what would $\text{PC}_2$ be?

Surely, it must be orthogonal to $\text{PC}_1$, but is that choice unique?

How can you differentiate between $\text{PC}_2^A=[-1,1,-1,1,-1,1,-1,1,-1,1]$ and $\text{PC}_2^B=[1,1,1,1,1,-1,-1,-1,-1,-1]$? Both are orthogonal to $\text{PC}_1$ after all.

See? The choice of $\text{PC}_2$ is not automatic given $\text{PC}_1$. In fact, it amounts to the same procedure to retrieve $\text{PC}_1$, after you remove the effect of $\text{PC}_1$ from data.

Answer 2

How about a more intuitive approach: You're thinking in 2D. The last component has no choice: it's determined by all of the other components. But intermediate components do have a choice. In the 2D case, the second choice is also the last choice, and therefore has no options: it must point in the direction that's determined by the prior choice.

Imagine a cloud of 3D points in space in front of you. Imagine your first component is a wooden rod that floats in space where you place it. You align this rod so that it lines up with the cloud's longest axis. Then you drill a hole in the side of this rod and place a second rod into it at a right angle. You can then rotate the original rod, pivoting the second rod through 360 degrees.

You can't make the second rod point in any direction, but you can pivot it through the circle perpendicular to the first rod. Your first rod constrains the second, but it does not absolutely determine where it points: you still have 360 degrees of rotation.

Now drill another hold at a right angle to the first two rods and insert your last rod. This one has no adjustability: your choices for the first two rods has totally constrained it instead of partially constrained it.

In higher dimensions, this is how it works. Your first component can point in any direction, your second component can point in any direction but it is constrained to a lower-dimensional range of choices than the first component. (Because you insist that the second must be perpendicular to the first.) The third is constrained by the first two -- it must be perpendicular to both, which cuts its choices down by two dimensions -- and so on.