4
$\begingroup$

I've got a problem choosing the right model. I have a model with various variables (covariables and dummy variables). I was trying to find the best size for this model, so I first started by comparing different models with AIC. From this it followed, that the minimum AIC was reached when allowing all variables to stay in the model (with the whole bunch to interact with all dummies). When I compute the summary of the model, all effects are absolutely not significant and the standard errors are very high. I was a bit confused, when comparing the "best" (on AIC) model with a smaller model with any interaction. The smaller model had small standard errors and nice p-values... But the AIC is higher compared to the big model. What might be the problem? Overspecification?

I really need help in this, because I have absolutely no idea how to handle this!

Thanks a lot

$\endgroup$
1
  • $\begingroup$ Thanks! This is a good hint! I'll keep myself busy in studying cross validation! I know the BIC criteria, but unfortunately it doesn't work with glms (I am using R) and I've heard it should not be used if you are not using a OLS method. Might be a rumor... At last a personal question: What would you recommend to use? AIC or Cross Validation? $\endgroup$
    – MarkDollar
    May 9, 2011 at 19:07

1 Answer 1

4
$\begingroup$

The AIC and standard error measure different things, and if you are trying to minimize standard error, a cross-validation approach may be better to use. Another alternative is the Bayesian information criterion (BIC), which is more parsimonious than the AIC.

Also, here's a good article comparing the relations between various evaluation metric for supervised machine learning: Data mining in metric space: an empirical analysis of supervised learning performance criteria.

$\endgroup$
3
  • $\begingroup$ As I understand it, AIC and cross validation (at least, leave-one-out cross validation) are asymptotically equivalent, therefore your suggesting that the OP employ cross validation boils down to ignoring the uncross-validated SE and go with the AIC instead. $\endgroup$ May 10, 2011 at 2:33
  • 3
    $\begingroup$ I'd also suggest that your statement that BIC is more parsimonious than AIC needs to be expanded to clarify that BIC will typically choose simpler models than AIC, but that this by no means implies that the models that BIC chooses are "better". BIC fundamentally assumes that the "true" model is amongst the models under consideration, which is probably a stretch for most circumstances. AIC on the other hand simply attempts to minimize future error prediction (hence the connection to cross validation) and makes no claims regarding the "truth" of the selected model. $\endgroup$ May 10, 2011 at 2:36
  • $\begingroup$ Thanks! This is a good hint! I'll keep myself busy in studying cross validation! I know the BIC criteria, but unfortunately it doesn't work with glms (I am using R) and I've heard it should not be used if you are not using a OLS method. Might be a rumor... At last a personal question: What would you recommend to use? AIC or Cross Validation? $\endgroup$
    – MarkDollar
    May 11, 2011 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.