I am performing model evaluation (via maximum likelihood) on several datasets with a number of different classes of models.
I have reasons to suspect that a specific class of models is (much) more flexible than the others, and therefore measures such as AIC might favour this more flexible class due to the risk of over-fitting. One of the reasons is that, for example, when I switch to BIC results change dramatically (although it is fair to say that BIC might be over-penalizing model complexity).
Therefore, I performed a 10-fold cross validation on stratified data ($9/10$-th training, $1/10$-th test), and evaluated the models using the sum (or average) log likelihood on the hold out test data, as an estimate of model out-of-sample performance.
To my surprise, the model class that I thought was more flexible(*) won the model comparison under 10-fold CV.
- Should I conclude that there is no over-fitting going on, notwithstanding the mixed evidence from other metrics (e.g., BIC) and prior information?
- Are there other tests you would perform to verify that this model class is (not) winning out of additional flexibility?
(*) Edit: not just more flexible, but excessively more flexible.