The idea of PCA is to find the directions (in high dimensional space) in which the essential structures (with regard to large variance, scatter) of the data lie. The assumption is that original features(variables) have a linear relationship. So, correlated original features(variables) are captured by PCA.
The steps of PCA are the following:
- Features are centred (but the direction does not change).
- The covariance matrix
S (K x K)is calculated (it is symmetrical).
- The eigenvalue and eigenvector are calculated. Normalized eigenvectors correspond to loadings (weights).
- Principal components i.e. scores, are calculated. Scores are weighted sums of the observations on the original features. So represented by linear combination, where principal components (
PC 1, PC 2 ... PC K) are orthogonal (because the covariance matrix is symmetrical).
- In the end you have to sort eigenvalues (variance) according to size and select principal components accordingly.
Now forget about everything I said before and assume that you've found
PC 1(with regard to highest variance). As you know, the next principal component (i.e. P
C 2) must be orthogonal to
PC 1. So we automatically know the direction of
PC 2, right? Since my space is
K dimensional I will take the next principal component (
PC 3) so that it is orthogonal to the first and second principal components, right? etc.
Now, could I say that if I had only determined the direction of the first principal component, all other direction of the principal components would be determined automatically?