While reading An Introduction to Statistical Learning, I stumbled across the following (p. 210):

[...] the model containing all of the predictors will always have the smallest $RSS$ and the largest $R^2$ , since these quantities are related to the training error.

The text talks about feature (input) selection in regressive models. The reference is the following (p. 205):

The problem is that a low $RSS$ or a high $R^2$ indicates a model with a low training error, whereas we wish to choose a model that has a low test error.

I don't understand why the authors relate those error measures to the training dataset, since we can calculate $RSS$ or $R^2$ on a test dataset (given that I have enough data) too.

Is there any reason for this distinction?

  • 1
    $\begingroup$ it would be helpful if you gave a larger context, but yes they do mean training RSS, and there is nothing to stop you applying RSS or R^2 on test data set ( and they discuss various approaches to estimating test performance later in the book) $\endgroup$ – seanv507 Jul 3 '16 at 18:41
  • $\begingroup$ @seanv507 Well, the context is selection of inputs (subset feature/variable selection) in linear model selection. Still, the authors say that >the same concepts apply to other methods. In any case, I was just wondering if there's theory or reason that I don't know, because they relate $RSS$ + $R^2$ to training only. In my understanding, the main thing is to validate and test (using any measure of error) with other data, if available, and that was it. $\endgroup$ – Luis Jul 4 '16 at 10:47
  • $\begingroup$ The authors use RSS often to mean training RSS (as in your quote), but do also talk about test RSS eg Q.4 on page 120-21) $\endgroup$ – seanv507 Jul 4 '16 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.