Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm doing some exploratory modelling on a data set with 29 covariates and an additional 11 variables that are of interest to my research question.

My strategy is to develop a model with a subset of the vector of covariates, and then evaluate model significance with the additional 11 research variables.

Irrespective of the specific modelling strategy, I'm afraid that I'm not sure how to deal with model uncertainty stemming from interpreting the AIC and BIC values I'm seeing.

In other words, if I calculate BIC for 5000 models, and find that the BIC is minimized (let's say = -1711) in a linear model with the variables A, B, C, and that the model with second-lowest BIC (let's say, = -1709) has the variables, A, B, C, and D -- how do I know that the difference between these two models is sufficient to justify excluding variable D from consideration? And how do I extend that examination of differences to consider the remaining 4,998 models?

Thanks in advance for any help you may be able to provide.

share|improve this question
    
As a rule of thumb, models with AIC differences $\leq 2$ are indistinguishable. See the wikipedia article on the AIC here: en.wikipedia.org/wiki/Akaike_information_criterion. –  Néstor Mar 6 '13 at 22:32
    
Thanks @Néstor for the helpful link. So I can use exp((AICmin−AICi)/2) as weights to calculate a multimodel? What are the implications of doing this with ~all~ available candidate models vs. only those where exp((AICmin−AICi)/2) > alpha? –  dubhousing Mar 7 '13 at 18:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.