3
$\begingroup$

How much is overfitting? For example, results on seen data is between 1 to 15% better than unseen data. Is there a range of value for example 2% where it is considered normal and not overfitting?

Also, Is there different range of value for different application? For example, maybe in information retrieval, 1 to 10% is normal, but maybe in image processing, 1 to 15% is normal.

$\endgroup$
1
$\begingroup$

Interesting question. I'm not familiar with such a definition.

You can evaluate that using cross validation.

  1. Do cross validation for X times 1.1 In each time measure the accuracy on both the training set and test set
  2. Test whether the train accuracy and test accuracy are likely to come from the same distribution (e.g. using Two-Sample t-Test for Equal Means)

Note that this procedure ignores the model complexity. If your model size is tiny with respect to the dataset, it cannot encode much of it. If you see a significant difference between the train and test accuracies there might be another problem there. Example of such a problem is a difference between your train and test datasets.

On the other hand, if you have a huge model and the accuracies are similar, you still might have over fitted but also failed to exploit that overfitting. Example of such a case is a large random forest while a one of its trees contributes most predictive power.

$\endgroup$
1
$\begingroup$

There are no hard-and-fast rules about what constitutes "over-fitting".

When using regression, heuristics are sometimes given in terms of the ratio of the sample size to the number of parameters, rather than the difference in predictive accuracy in-sample versus out-of-sample. E.g., in a regression context, I recall reading both 3 and 5 observations per observation being the minimum.

In predictive applications, the goal is to improve your out-of-sample prediction, so if you have a model that is 20% worse out of sample than in-sample, but still predicts better than a model that is only 1% better out-of-sample than in-sample, you should going to prefer the model that is over-fitted. Of course, in such a situation you will also want to find ways of reducing the over-fitting, while keeping the out-of-sample prediction constant.

$\endgroup$
1
$\begingroup$

Another suggestion: You could also take a look at the OOB error in Random Forest. This OOB error should actually be worse on the training data then the error on the test data (since on the training data only trees are used which haven't been trained on the observation and therefor these are less trees).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.