5
$\begingroup$

In principal components analysis, the principal components are chosen according to three criteria. The first component is chosen to be the direction in the data with greatest variance. The second is chosen to be the direction with greatest variance GIVEN that it is orthogonal to the first.

I am wondering if there is an intuitive way to understand this without having to resort to the proof, of which is hard for me to extract intuition from? Thanks.

$\endgroup$
6
  • 1
    $\begingroup$ If it weren't orthogonal (i.e. independent), it would explain variance that is already captured by the first component! Also see the great answers here: stats.stackexchange.com/questions/2691/… $\endgroup$ Commented Oct 6, 2017 at 0:51
  • $\begingroup$ Is that the same as saying that we are creating principal components that are correlated, and hence may introduce a collinearity issue when trying to regress on these summary/principal component variables? $\endgroup$
    – user321627
    Commented Oct 6, 2017 at 0:57
  • $\begingroup$ Do you have experience applying PCA to data sets? If so, in those cases, what lead you to use PCA? $\endgroup$
    – user795305
    Commented Oct 6, 2017 at 1:01
  • 1
    $\begingroup$ They would indeed be correlated if they were not orthogonal. The original variables may or may not be too strongly correlated leading to multicollinearity, which is one of the reasons one might use PCA prior to regression (PCR). If regression is your goal, there exist other alternatives you may want to consider, such as PLS or ridge regression. What is the purpose of your research? $\endgroup$ Commented Oct 6, 2017 at 1:14
  • 1
    $\begingroup$ If you didn't include the orthogonality constraint, the solution to the second optimization problem obviously would be the first principal component, because nothing would have changed. What constraint, then, do you propose erecting in place of orthogonality? (I don't see how your question could be answered without knowing what you have in mind.) $\endgroup$
    – whuber
    Commented Oct 6, 2017 at 14:36

2 Answers 2

6
$\begingroup$

Say you fit the first component by maximizing the variance. Now, try to fit the second component by again maximizing the variance, but don't enforce any orthogonality constraints. You'll find that the "second" component in this case is identical to the first, up to an arbitrary sign flip (i.e. it will point in either the same or the opposite direction as the first component). This is because the optimization problem hasn't changed at all. Solving for the "second" component in this case is simply asking for the first component all over again.

$\endgroup$
1
  • $\begingroup$ How about we constrain on the condition a bit, say except the direction of the first component? What is the result, then? Is that the orthogonal one $\endgroup$
    – Catbuilts
    Commented Oct 13, 2020 at 19:11
1
$\begingroup$

If the components weren't orthogonal then you could project the second component onto the first. Look at a and b vectors, they're not orthogonal. So, if you step in the direction of a vector, you actually move both up and right. Hence, these would be correlated.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.