3
$\begingroup$

In their book The Elements of Statistical Learning, Friedman and coll. talk about the in sample error $Err_{in}$ (p. 229): $$Err_{in} = \frac{1}{n}\sum_{i=1}^n\mathbb{E}\left[\left(Y_i-\hat{f}\left( x_i\right)\right)^2\right] $$ there exists an unbiased estimator of this error in case of linear regression which is $$Err_{in}=\text{training_error} + 2\frac{d}{n}\hat{\sigma_{\epsilon}}^2$$ with $d$ the dimension, $n$ the number of observations and $\hat{\sigma_{\epsilon}}^2$ the unbiased estimator of the residuals.

My question is how do you use this criterion ?

$\endgroup$
2
$\begingroup$

The idea is that you can compare this "in sample error" for different models.

So you fit a number of models (with more or less main effects and/or interactions) that you think might be of interest. Next you calculate the in sample error for each of these (fitted) models, and you find the model with the smallest in sample error: this is supposedly the 'best' model (wrt this criterion).

In practice, many people use crossvalidation to achieve model comparison instead of this in sample error, but different criteria may give different results (which may or may not be wanted).

$\endgroup$
  • $\begingroup$ Thanks you. Thus i can use cross validation error or in sample error to choose a model ? Any idea if one is better than the other ? $\endgroup$ – grant Nov 4 '11 at 13:01
  • $\begingroup$ CV error <-> IS error: yes. Which is better? That will greatly depend on your goal. It may not even be so that either serves you best (e.g. if false negatives are worse to you than false positives). I seem to recall that Friedman et al are rather advocates of CV, but even then you have a ton of criteria for different goals (missclassification, AUC, deviance, ...) You have to figure out which one is the best for your goal. $\endgroup$ – Nick Sabbe Nov 4 '11 at 13:32
  • $\begingroup$ Thanks you. machine learning is mystic science sometimes $\endgroup$ – grant Nov 4 '11 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.