Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
How much is overfitting? For example, results on seen data is between 1 to 15% better than unseen data. Is there a range of value for example 2% where it is considered normal and not overfitting?
Also, Is there different range of value for different application? For example, maybe in information retrieval, 1 to 10% is normal, but maybe in image processing, 1 to 15% is normal.
There are no hard-and-fast rules about what constitutes "over-fitting".
When using regression, heuristics are sometimes given in terms of the ratio of the sample size to the number of parameters, rather than the difference in predictive accuracy in-sample versus out-of-sample. E.g., in a regression context, I recall reading both 3 and 5 observations per observation being the minimum.
In predictive applications, the goal is to improve your out-of-sample prediction, so if you have a model that is 20% worse out of sample than in-sample, but still predicts better than a model that is only 1% better out-of-sample than in-sample, you should going to prefer the model that is over-fitted. Of course, in such a situation you will also want to find ways of reducing the over-fitting, while keeping the out-of-sample prediction constant.
Interesting question. I'm not familiar with such a definition.
You can evaluate that using cross validation.
Note that this procedure ignores the model complexity. If your model size is tiny with respect to the dataset, it cannot encode much of it. If you see a significant difference between the train and test accuracies there might be another problem there. Example of such a problem is a difference between your train and test datasets.
On the other hand, if you have a huge model and the accuracies are similar, you still might have over fitted but also failed to exploit that overfitting. Example of such a case is a large random forest while a one of its trees contributes most predictive power.
Another suggestion: You could also take a look at the OOB error in Random Forest. This OOB error should actually be worse on the training data then the error on the test data (since on the training data only trees are used which haven't been trained on the observation and therefor these are less trees).
