Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If I have a data, and I run a classification (let's say random forest on this data) with cross validation (let's say 5-folds), could I conclude that there is no over fitting in my method?

share|improve this question
up vote 12 down vote accepted

Not at all. However, cross validation helps you to assess by how much your method overfits.

For instance, if your training data R-squared of a regression is 0.50 and the crossvalidated R-squared is 0.48, you hardly have any overfitting and you feel good. On the other hand, if the crossvalidated R-squared is only 0.3 here, then a considerable part of your model performance comes due to overfitting and not from true relationships. In such a case you can either accept a lower performance or try different modelling strategies with less overfitting.

share|improve this answer
3  
I think this answer is correct in spirit, but I disagree with the characterization of over fitting in the second paragraph. I don't believe that over fitting occurs when train error - test error > some bound, instead, I would characterize over fitting as a situation where increasing the complexity of the model slightly tends to increase the hold out error. Requiring that your train and test errors are comparable will often result in very underfit models. – Matthew Drury Feb 2 at 16:52

Cross-Validation is a good, but not perfect, technique to minimize over-fitting.

Cross-Validation will not perform well to outside data if the data you do have is not representative of the data you'll be trying to predict!

Here are two concrete situations when cross-validation has flaws:

  • You are using the past to predict the future: it is often a big assumption to assume that past observations will come from the same population with the same distribution as future observations. Cross-validating on a data set drawn from the past won't protect against this.
  • There is a bias in the data you collect: the data you observe is systematically different from the data you don't observed. For example, we know about respondent bias in those who chose to take a survey.
share|improve this answer
    
Having your dataset not being a poor representation of the true population is generally considered a separate issue of over fitting. Of course, it is correct that cross-validation does not address them. – Cliff AB Feb 3 at 1:49

Also I can recomend these videos from the Stanford course in Statistical learning. These videos goes in quite depth regarding how to use cross-valudation effectively.

Cross-Validation and the Bootstrap (14:01)

K-fold Cross-Validation (13:33)

Cross-Validation: The Right and Wrong Ways (10:07)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.