Model selection with AIC and model efficiency

Starting from an "a priori" set of models based on my knowledge about potential relations between my dependent variable and the independent variables considered, I use AIC to find best models. More specifically I calculate Akaike weights then Evidence Ratio (ER) and consider that models with a ER < 2 are equally likely. But the same problem remain each time I do that. I selected the best models from a set of them, but I don't know if those models are efficient to predict (or at least represent) my data. I can have selected the best element(s) of the list of the worst models.

I do not use $R^2$ in model selection because of the fact that including more variables generally increase $R^2$ value.

But ! When the selection by AIC is done and I can consider that models with a ER < 2 are equally likely. Do you find it is correct to calculate $R^2$ or pseudo-$R^2$ for the best "set of models" in order to have an idea of the representativeness of those models and use this value to select the more efficient model?

Note: Thanks for suggestions about cross-validation, I will try this. Unfortunatly, I do not have an external dataset to test my models.

• I'm confused: your first paragraph suggests that you find a set of models somehow, though you don't specify how. Next, you find the best of those models through ER. Since you do not specify how you got to your first set, it's hard to answer that part. As per R2: there is much debate on its usefulness. If it is an option for you, use cross validation and a real predictive measure like missclassification rate or AUC. – Nick Sabbe Aug 25 '11 at 7:35

I agree cross validation + good measure of error should be the way to go. Maybe you can too use ridge and lasso instead of AIC to extract strong predictors in order to get a robust model.

• ... and don't forget to validate the optimal model with new independent test data (or an outer cross validation loop) – cbeleites supports Monica Aug 25 '11 at 12:43
• Related to that good comment, note that using AIC is in essence another way to use P-values to drive model selection, hence the procedure has most of the problems of stepwise variable selection. – Frank Harrell Aug 25 '11 at 19:33
• (+1) Welcome to our site! Thanks for answering. – whuber Aug 28 '11 at 22:05
• Cross-validation is essentially equivalent to AIC or GCV in finite dimensions (fixed p), although the connection hasn't been well explored in high or ultra-high dimensions. Even with ridge or LASSO, you will still have to determine the tuning parameter by resorting back to AIC, GCV, or BIC. – XGS Sep 18 '16 at 22:15