1
$\begingroup$

Are there heuristics/techniques that can be employed to guard against overfitting of a model that do not require/employ human inspection of the learned model's performance.

So for example, they would look at the performance of a predictor and come up with a score of how likely it is that the model has been overfit.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Compare training set performance to independent test set performance. $\endgroup$
    – Marc Claesen
    Commented Jun 25, 2015 at 6:52
  • $\begingroup$ @MarcClaesen, mmm, so say I restrict ourselves to the classical cross validation methods -- we will partition our data into training and test. Can we come up with a measure of overfitting? $\endgroup$
    – user1172468
    Commented Jun 25, 2015 at 6:54
  • 1
    $\begingroup$ Yes, assess performance based on all training and test set pairs during cross-validation. Then at the end you can compare both results. If they are comparable, there was no overfitting. $\endgroup$
    – Marc Claesen
    Commented Jun 25, 2015 at 6:56
  • $\begingroup$ @MarcClaesen cool -- so could we formulate that into a formulae? And is there any extensive research on such measures. $\endgroup$
    – user1172468
    Commented Jun 25, 2015 at 6:59

1 Answer 1

Reset to default
2
$\begingroup$

The most direct way to assess overfitting is by looking at any performance differences on the training set compared to an independent test set. This can be automated easily.

In cross-validation, you can assess performance based on all training and test set pairs. Then at the end you can compare the results of training set predictions and test set predictions. If they are comparable, there was no overfitting. If the training performance was far better than test, there was overfitting.

Note that if you're going to perform model selection using cross-validation, you are unlikely to end up with a model that overfits, because that model wouldn't have good cross-validation performance anyway.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ (+1) But the last paragraph requires some qualification: performance estimates obtained by cross-validation are statistics like any other, subject to sampling variation; using them to tune a single hyperparameter (as with e.g. LASSO) may not (often) lead to over-fitting, but using them to select one from many models, in a best-subsets or stepwise style approach, is liable to. See Can you overfit by training machine learning algorithms using CV/Bootstrap?. $\endgroup$
    – Scortchi ♦
    Commented Jun 25, 2015 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.