I was told that cross-validation and the Bayesian method can overcome the overfitting problem. I was told that comparison of models in a Bayesian way is actually doing cross-validation... What is the explanation?
I think this is a two part question:
- How do a) cross-validation and b) Bayesian methods overcome overfitting.
- How are they similar.
Overfitting is essentially producing a model tailored to a specific dataset. Like learning a string of ABCABAB answers in a multiple choice test. Change the questions slightly and your model is shot.
Cross-validation tests whether your model is shot by training it on part of the data and testing it on an unseen part. As there are multiple ways to split into train and test and you can average the test scores over these.
The Bayesian method works by using probabilities, which always sum to one. If you have a big (complex) class of models you'd like to try out then you have to spread your prior probability thinly over them. A bigger class of models is more likely to hold a likelihood function which fits the data very well. But as posterior is $\propto$ likelihood $\times$ prior your inference is trading off finding a well-fitting likelihood against thinly spreading your prior probability. You then typically average over the posterior rather than focussing on the single best fitting likelihood function.
As for how they are similar I'd suggest Gelman et al's Bayesian Data Analysis. There's a chapter on information criteria. They suggest the series expansion of WAIC is similar to that of LOO-CV (though I have not explored this in detail).