Can anyone tell me how to judge if a supervised machine learning model is overfitting or not? If I don't have an external validation dataset, I want to know if I can use ROC of 10 fold cross validation to explain overfitting. If I have an external validation dataset, what should I do next?


3 Answers 3


In short: by validating your model. The main reason of validation is to assert no overfit occurs and to estimate generalized model performance.


First let us look at what overfitting actually is. Models are normally trained to fit a dataset by minimizing some loss function on a training set. There is however a limit where minimizing this training error will no longer benefit the models true performance, but only minimize the error on the specific set of data. This essentially means that the model has been too tightly fitted to the specific data points in the training set, trying to model patterns in the data originating from noise. This concept is called overfit. An example of overfit is displayed below where you see the training set in black and a larger set from the actual population in the background. In this figure you can see that the blue model is too tightly fitted to the training set, modeling the underlying noise. enter image description here

In order to judge if a model is overfitted or not, we need to estimate the generalized error (or performance) that the model will have on future data and compare it to our performance on the training set. Estimating this error can be done in several different ways.

Dataset split

The most straightforward approach to estimating the generalized performance is to partition the dataset into three parts, a training set, a validation set and a test set. The training set is used for training the model to fit the data, the validation set is used to measure differences in performance between models in order to select the best one and the test set to assert that the model selection process does not overfit to the first two sets.

To estimate the amount of overfit simply evaluate your metrics of interest on the test set as a last step and compare it to your performance on the training set. You mention ROC but in my opinion you should also look at other metrics such as for example brier score or a calibration plot to ensure model performance. This is of course depending on your problem. There are a lot of metrics but this is besides the point here.

This method is very common and respected but it puts a big demand on availability of data. If your dataset is too small you will most probably lose a lot of performance and your results will be biased on the split.


One way to get around wasting a large part of the data to validation and test is to use cross-validation (CV) which estimates the generalized performance using the same data as is used to train the model. The idea behind cross-validation is to split the dataset up into a certain number of subsets, and then use each of these subsets as held out test sets in turn while using the rest of the data to train the model. Averaging the metric over all the folds will give you an estimate of the model performance. The final model is then generally trained using all data.

However, the CV estimate is not unbiased. But the more folds you use the smaller the bias but then you get larger variance instead.

As in the dataset split we get an estimate of the model performance and to estimate the overfit you simply compare the metrics from your CV with the ones acquired from evaluating the metrics on your training set.


The idea behind bootstrap is similar to CV but instead of splitting the dataset into parts we introduce randomness in the training by drawing training sets from the whole dataset repeatedly with replacement and performing the full training phase on each of these bootstrap samples.

The simplest form of bootstrap validation simply evaluates the metrics on the samples not found in the training set (i.e. the ones left out) and average over all repeats.

This method will give you an estimate of model performance which in most cases are less biased than CV. Again, comparing it with your training set performance and you get the overfit.

There are ways to improve the bootstrap validation. The .632+ method is known to give better, more robust estimates of the generalized model performance, taking overfit into account. (If you're interested the original article is a good read: Improvements on Cross-Validation: The 632+ Bootstrap Method)

I hope this answers your question. If you are interested in model validation I recommend reading the part on validation in the book The elements of statistical learning: data mining, inference and prediction which is freely available online.

  • 4
    $\begingroup$ Note that your terminology of validating vs. testing is not followed in all fields. E.g. in my field (analytical chemistry) validation is a procedure that should prove that the model works well (and measure how well it works). It is done with the final model, no further changes are allowed afterwards (or, if you do so, you need to validate again with independent data). So I'd call your validation set an "inner test set" or "optimization test set". The "outer" test data does not prevent overfitting, but it may be used to measure the extent of overfitting. $\endgroup$ Jan 8, 2014 at 13:16
  • $\begingroup$ Ok, I don't have experience in your field. Thanks for the clarification. It is probably the same in other fields as well. I simply used the terminology used in the book I linked to in the end. I hope it is not too confusing. $\endgroup$
    – while
    Jan 8, 2014 at 13:21

Here's how you can estimate the extent of overfitting:

  • Get an internal error estimate. Either resubstitutio (= predict training data), or if you do an inner cross "validation" to optimize hyperparameters, also that measure would be of interest.
  • Get an independent test set error estimate. Usually, resampling (iterated cross validation or out-of-bootstrap* is recommended. But you need to be careful that no data leaks occur. I.e. the resampling loop must recalculate all steps that have calculations spanning more than one case. That includes pre-processing steps such as centering, scaling, etc. Also, make sure you split at the highest level if you have a "hierarchical" (also known as "clustered) data structure such as repeated measurements of e.g. the same patient (=> resample patients).
  • Then compare how much better the "inner" error estimate looks than the independent one.

Here's an example:
Trefferrate = hit rate (% correct classified), Variablenzahl = number of variables (= model complexity)
Symbols: . resubstitution, + internal leave-one-out estimate of hyperparameter optimizer, o outer cross validation independent at patient level

This works with ROC, or performance measures such as Brier's score, sensitivity, specificity, ...

* I don't recommend .632 or .632+ bootstrap here: they already mix in resubstitution error: you can anyways calculate them later from your resubstitution and out-of-bootstap estimates.


The overfitting is simply the direct consequence of considering the statistical parameters, and therefore the results obtained, as a useful information without checking that them was not obtained in a random way. Therefore, in order to estimate the presence of overfitting we have to use the algorithm on a database equivalent to the real one but with randomly generated values, repeating this operation many times we can estimate the probability of obtaining equal or better results in a random way. If this probability is high, we are most likely in an overfitting situation. For example, the probability that a fourth-degree polynomial has a correlation of 1 with 5 random points on a plane is 100%, so this correlation is useless and we are in an overfitting situation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.