# How do I know if the differences in ICs among candidate models are significant?

I'm doing some exploratory modelling on a data set with 29 covariates and an additional 11 variables that are of interest to my research question.

My strategy is to develop a model with a subset of the vector of covariates, and then evaluate model significance with the additional 11 research variables.

Irrespective of the specific modelling strategy, I'm afraid that I'm not sure how to deal with model uncertainty stemming from interpreting the AIC and BIC values I'm seeing.

In other words, if I calculate BIC for 5000 models, and find that the BIC is minimized (let's say = -1711) in a linear model with the variables A, B, C, and that the model with second-lowest BIC (let's say, = -1709) has the variables, A, B, C, and D -- how do I know that the difference between these two models is sufficient to justify excluding variable D from consideration? And how do I extend that examination of differences to consider the remaining 4,998 models?

As a rule of thumb, models with AIC differences $\leq 2$ are indistinguishable. See the wikipedia article on the AIC here: en.wikipedia.org/wiki/Akaike_information_criterion. –  Néstor Mar 6 '13 at 22:32