If I have a data, and I run a classification (let's say random forest on this data) with cross validation (let's say 5-folds), could I conclude that there is no over fitting in my method?
Not at all. However, cross validation helps you to assess by how much your method overfits.
For instance, if your training data R-squared of a regression is 0.50 and the crossvalidated R-squared is 0.48, you hardly have any overfitting and you feel good. On the other hand, if the crossvalidated R-squared is only 0.3 here, then a considerable part of your model performance comes due to overfitting and not from true relationships. In such a case you can either accept a lower performance or try different modelling strategies with less overfitting.
Cross-Validation is a good, but not perfect, technique to minimize over-fitting.
Cross-Validation will not perform well to outside data if the data you do have is not representative of the data you'll be trying to predict!
Here are two concrete situations when cross-validation has flaws:
- You are using the past to predict the future: it is often a big assumption to assume that past observations will come from the same population with the same distribution as future observations. Cross-validating on a data set drawn from the past won't protect against this.
- There is a bias in the data you collect: the data you observe is systematically different from the data you don't observed. For example, we know about respondent bias in those who chose to take a survey.
Also I can recomend these videos from the Stanford course in Statistical learning. These videos goes in quite depth regarding how to use cross-valudation effectively.