4
$\begingroup$

In their book The Elements of Statistical Learning, Friedman and coll. talk about the in sample error $Err_{in}$ (p. 229): $$Err_{in} = \frac{1}{n}\sum_{i=1}^n\mathbb{E}\left[\left(Y_i-\hat{f}\left( x_i\right)\right)^2\right] $$ there exists an unbiased estimator of this error in case of linear regression which is $$Err_{in}=\text{training_error} + 2\frac{d}{n}\hat{\sigma_{\epsilon}}^2$$ with $d$ the dimension, $n$ the number of observations and $\hat{\sigma_{\epsilon}}^2$ the unbiased estimator of the residuals.

My question is how do you use this criterion ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The idea is that you can compare this "in sample error" for different models.

So you fit a number of models (with more or less main effects and/or interactions) that you think might be of interest. Next you calculate the in sample error for each of these (fitted) models, and you find the model with the smallest in sample error: this is supposedly the 'best' model (wrt this criterion).

In practice, many people use crossvalidation to achieve model comparison instead of this in sample error, but different criteria may give different results (which may or may not be wanted).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks you. Thus i can use cross validation error or in sample error to choose a model ? Any idea if one is better than the other ? $\endgroup$
    – grant
    Commented Nov 4, 2011 at 13:01
  • $\begingroup$ CV error <-> IS error: yes. Which is better? That will greatly depend on your goal. It may not even be so that either serves you best (e.g. if false negatives are worse to you than false positives). I seem to recall that Friedman et al are rather advocates of CV, but even then you have a ton of criteria for different goals (missclassification, AUC, deviance, ...) You have to figure out which one is the best for your goal. $\endgroup$
    – Nick Sabbe
    Commented Nov 4, 2011 at 13:32
  • $\begingroup$ Thanks you. machine learning is mystic science sometimes $\endgroup$
    – grant
    Commented Nov 4, 2011 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.