0
$\begingroup$

Are there heuristics/techniques that can be employed to guard against overfitting of a model that do not require/employ human inspection of the learned model's performance.

So for example, they would look at the performance of a predictor and come up with a score of how likely it is that the model has been overfit.

$\endgroup$
  • 1
    $\begingroup$ Compare training set performance to independent test set performance. $\endgroup$ – Marc Claesen Jun 25 '15 at 6:52
  • $\begingroup$ @MarcClaesen, mmm, so say I restrict ourselves to the classical cross validation methods -- we will partition our data into training and test. Can we come up with a measure of overfitting? $\endgroup$ – user1172468 Jun 25 '15 at 6:54
  • 1
    $\begingroup$ Yes, assess performance based on all training and test set pairs during cross-validation. Then at the end you can compare both results. If they are comparable, there was no overfitting. $\endgroup$ – Marc Claesen Jun 25 '15 at 6:56
  • $\begingroup$ @MarcClaesen cool -- so could we formulate that into a formulae? And is there any extensive research on such measures. $\endgroup$ – user1172468 Jun 25 '15 at 6:59
2
$\begingroup$

The most direct way to assess overfitting is by looking at any performance differences on the training set compared to an independent test set. This can be automated easily.

In cross-validation, you can assess performance based on all training and test set pairs. Then at the end you can compare the results of training set predictions and test set predictions. If they are comparable, there was no overfitting. If the training performance was far better than test, there was overfitting.

Note that if you're going to perform model selection using cross-validation, you are unlikely to end up with a model that overfits, because that model wouldn't have good cross-validation performance anyway.

$\endgroup$
  • 2
    $\begingroup$ (+1) But the last paragraph requires some qualification: performance estimates obtained by cross-validation are statistics like any other, subject to sampling variation; using them to tune a single hyperparameter (as with e.g. LASSO) may not (often) lead to over-fitting, but using them to select one from many models, in a best-subsets or stepwise style approach, is liable to. See Can you overfit by training machine learning algorithms using CV/Bootstrap?. $\endgroup$ – Scortchi - Reinstate Monica Jun 25 '15 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.