If I have a data, and I run a classification (let's say random forest on this data) with cross validation (let's say 5-folds), could I conclude that there is no over fitting in my method?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
Not at all. However, cross validation helps you to assess by how much your method overfits.
For instance, if your training data R-squared of a regression is 0.50 and the crossvalidated R-squared is 0.48, you hardly have any overfitting and you feel good. On the other hand, if the crossvalidated R-squared is only 0.3 here, then a considerable part of your model performance comes due to overfitting and not from true relationships. In such a case you can either accept a lower performance or try different modelling strategies with less overfitting.
Cross-Validation is a good, but not perfect, technique to minimize over-fitting.
Cross-Validation will not perform well to outside data if the data you do have is not representative of the data you'll be trying to predict!
Here are two concrete situations when cross-validation has flaws:
Also I can recomend these videos from the Stanford course in Statistical learning. These videos goes in quite depth regarding how to use cross-valudation effectively.