3
$\begingroup$

Starting from an "a priori" set of models based on my knowledge about potential relations between my dependent variable and the independent variables considered, I use AIC to find best models. More specifically I calculate Akaike weights then Evidence Ratio (ER) and consider that models with a ER < 2 are equally likely. But the same problem remain each time I do that. I selected the best models from a set of them, but I don't know if those models are efficient to predict (or at least represent) my data. I can have selected the best element(s) of the list of the worst models.

I do not use $R^2$ in model selection because of the fact that including more variables generally increase $R^2$ value.

But ! When the selection by AIC is done and I can consider that models with a ER < 2 are equally likely. Do you find it is correct to calculate $R^2$ or pseudo-$R^2$ for the best "set of models" in order to have an idea of the representativeness of those models and use this value to select the more efficient model?

I would be glad to hear your opinions about this!

Note: Thanks for suggestions about cross-validation, I will try this. Unfortunatly, I do not have an external dataset to test my models.

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ I'm confused: your first paragraph suggests that you find a set of models somehow, though you don't specify how. Next, you find the best of those models through ER. Since you do not specify how you got to your first set, it's hard to answer that part. As per R2: there is much debate on its usefulness. If it is an option for you, use cross validation and a real predictive measure like missclassification rate or AUC. $\endgroup$
    – Nick Sabbe
    Commented Aug 25, 2011 at 7:35

1 Answer 1

Reset to default
3
$\begingroup$

I agree cross validation + good measure of error should be the way to go. Maybe you can too use ridge and lasso instead of AIC to extract strong predictors in order to get a robust model.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ ... and don't forget to validate the optimal model with new independent test data (or an outer cross validation loop) $\endgroup$
    – cbeleites
    Commented Aug 25, 2011 at 12:43
  • 1
    $\begingroup$ Related to that good comment, note that using AIC is in essence another way to use P-values to drive model selection, hence the procedure has most of the problems of stepwise variable selection. $\endgroup$
    – Frank Harrell
    Commented Aug 25, 2011 at 19:33
  • $\begingroup$ (+1) Welcome to our site! Thanks for answering. $\endgroup$
    – whuber ♦
    Commented Aug 28, 2011 at 22:05
  • $\begingroup$ Cross-validation is essentially equivalent to AIC or GCV in finite dimensions (fixed p), although the connection hasn't been well explored in high or ultra-high dimensions. Even with ridge or LASSO, you will still have to determine the tuning parameter by resorting back to AIC, GCV, or BIC. $\endgroup$
    – XGS
    Commented Sep 18, 2016 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.