Is the RSS in a simple linear regression onto the largest principal component always less than or equal to the RSS of a simple regression onto the rest of the smaller principal components? I feel that this should be the case because the variation in the data is greatest along largest principal components, so I feel that there should be better fits along larger components. Is there a flaw in my reasoning, and if so, why? Thanks in advance.
As ISLR notes on pages 233-4, principal components regression is based on the hope that:
... the directions in which [the predictors] show the most variation are the directions that are associated with [outcome]. While this assumption is not guaranteed to be true, it often turns out to be a reasonable enough approximation to give good results.
So there can be no guarantee that RSS after regression onto the largest principal component will be less than that after regression onto smaller principal components. It might often be true in practice, but you can't count on it in general.