0
$\begingroup$

Is the RSS in a simple linear regression onto the largest principal component always less than or equal to the RSS of a simple regression onto the rest of the smaller principal components? I feel that this should be the case because the variation in the data is greatest along largest principal components, so I feel that there should be better fits along larger components. Is there a flaw in my reasoning, and if so, why? Thanks in advance.

$\endgroup$
  • 1
    $\begingroup$ Exactly which "principal components" are you referring to? Evidently this is a PCA, but does it include the response variable or not? And if it doesn't include the response variable, why should there be any reason to suppose there is some universal relationship (or even some meaningful restrictions) between the response variable and any subsets of the principal components? $\endgroup$ – whuber Mar 19 '17 at 18:18
0
$\begingroup$

As ISLR notes on pages 233-4, principal components regression is based on the hope that:

... the directions in which [the predictors] show the most variation are the directions that are associated with [outcome]. While this assumption is not guaranteed to be true, it often turns out to be a reasonable enough approximation to give good results.

So there can be no guarantee that RSS after regression onto the largest principal component will be less than that after regression onto smaller principal components. It might often be true in practice, but you can't count on it in general.

$\endgroup$
  • $\begingroup$ What is the justification behind this statement? I'd like to understand why this is the case. It seems very counterintuitive to me. $\endgroup$ – ping Mar 19 '17 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.