If I have a data, and I run a classification (let's say random forest on this data) with cross validation (let's say 5-folds), could I conclude that there is no over fitting in my method?
16
$\begingroup$
$\endgroup$
20
$\begingroup$
$\endgroup$
Not at all. However, cross validation helps you to assess by how much your method overfits.
For instance, if your training data R-squared of a regression is 0.50 and the crossvalidated R-squared is 0.48, you hardly have any overfitting and you feel good. On the other hand, if the crossvalidated R-squared is only 0.3 here, then a considerable part of your model performance comes due to overfitting and not from true relationships. In such a case you can either accept a lower performance or try different modelling strategies with less overfitting.
-
8$\begingroup$ I think this answer is correct in spirit, but I disagree with the characterization of over fitting in the second paragraph. I don't believe that over fitting occurs when train error - test error > some bound, instead, I would characterize over fitting as a situation where increasing the complexity of the model slightly tends to increase the hold out error. Requiring that your train and test errors are comparable will often result in very underfit models. $\endgroup$ – Matthew Drury Feb 2 '16 at 16:52
7
$\begingroup$
$\endgroup$
Cross-Validation is a good, but not perfect, technique to minimize over-fitting.
Cross-Validation will not perform well to outside data if the data you do have is not representative of the data you'll be trying to predict!
Here are two concrete situations when cross-validation has flaws:
- You are using the past to predict the future: it is often a big assumption to assume that past observations will come from the same population with the same distribution as future observations. Cross-validating on a data set drawn from the past won't protect against this.
- There is a bias in the data you collect: the data you observe is systematically different from the data you don't observed. For example, we know about respondent bias in those who chose to take a survey.
answered Feb 2 '16 at 20:57
-
3$\begingroup$ Having your dataset not being a poor representation of the true population is generally considered a separate issue of over fitting. Of course, it is correct that cross-validation does not address them. $\endgroup$ – Cliff AB Feb 3 '16 at 1:49
2
$\begingroup$
$\endgroup$
Also I can recomend these videos from the Stanford course in Statistical learning. These videos goes in quite depth regarding how to use cross-valudation effectively.
Cross-Validation and the Bootstrap (14:01)
