Model Selection: AIC/BIC and Cross-Validation gives different conclusion In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC, BIC ,Mallow's CP. But I am wondering about how should a researcher choose a suitable model selection method for a specific model?
According to my knowledge, people sometimes prefer AIC/BIC over Cross-Validation because of the computational efficiency of AIC/BIC or when the sample size is relatively small for cross-validation. In this thread, AIC is asymptotically equivalent to leave-1-out cross-validation (LOOCV)  , suggesting the large sample equivalence of AIC/BIC and Cross-Validation. This means that when the sample size grow (and ignoring computational issue), ICs and cross-validation should gives similar conclusions.
But for my project now, I tried to build a Logistic Regression model using a dataset with sample size ~8000. I want to choose the best subset of feature for my model. Let say, I have set of feature [A,B,C] versus [A,B,C,D,E,F] and would like to determine which set of feature will gives a better performance.
I face similar issue as Variable selection : combining AIC and Cross validation, when using AIC/BIC to evaluate my model, feature D,E,F turns out to be insignificant and the performance of [A,B,C](in terms of AIC/BIC) is better.
But when I use 10-fold cross-validation to evaluate model's performance. [A,B,C,D,E,F] gives a significantly better performance on validation set performance versus [A,B,C].
Therefore, I am wonder if there is some good way to choose model selection method, especially when different model selection method gives distinct conclusion.
 A: Maybe you should concentrate more on the methods that are intended precisely for feature selection, rather than model selection. Model selection methods like cross-validation or AIC try to compare models independently of how they differ (this is only approximately true, but should suffice here). Feature selection methods concentrate on comparing models that differ only by feature selection.
I have had good experience with e.g. random forest based feature selection, but there are many others, some of them more specialized like spike and slab.
Having said that, the results of those methods often contradict, especially in the less simple cases. Use the ones that are top in multiple methods as suggestions and you then select the model that works best for you w.r.t. other cost criteria like complexity, runtime...
A: AIC is asymptotically equivalent to leave-1-out cross-validation (LOOCV)

*

*It's not equivalent to 10-fold cross-validation, which is what you're comparing it to.

*It's only asymptotically equivalent, so the two methods don't always give the same answer, they're only approximately the same.

It's not really clear how you're doing the train/test/validation split when cross validating, so I can't really address your final question. Note, though, that the "best" model depends on the amount of data you're using. For example, a model with 5 features may perform well when trained on the full dataset, but could overfit when trained on only a subset in cross-validation.
A: Nothing strange in here.

*

*If all the model selection methods always gave the same results, we wouldn't have multiple criteria, but just pick arbitrary one.

*AIC and BIC explicitly penalize the number of parameters, cross-validation not, so again, it's not surprising that they suggest a model with fewer parameters (though nothing prohibits cross-validation from picking a model with fewer parameters).

