# Model selection using mean AIC for very huge data sets

I want to select a model which best performs for a very huge data set. However, the data set is too large to calculate a model within reasonable time.

If this is the case, is the following a reasonable approach: Fit each model to $n$ smaller random subsets of the original data set and calculate the mean AIC. Then, select the model with the lowest mean AIC.

• To be clear; are you fitting the same model to every subset? – Stijn Sep 6 '14 at 8:45
• Yes. I fit the same model to each subset. However, does this matter if $n$ is large? – Funkwecker Sep 8 '14 at 5:33
• I imagine it does as different models will give different AIC values, and I don't know about the behaviour of the mean AIC value. – Stijn Sep 8 '14 at 7:17
• What makes you need to select a model vs. fit a pre-specified model, ideally with penalization (shrinkage)? Note that the use of AIC for model selection is very similar to the highly disrespected stepwise regression. – Frank Harrell Sep 9 '15 at 12:39
• Can you elaborate on why AIC for model selection is disrespected? Thx. – Funkwecker Apr 7 '16 at 13:26

What you're trying to do sounds a lot like what the authors of this paper propose: http://arxiv.org/abs/1112.5016

Essentially what they do is bootstrapping on small subsets of their very large dataset and then average over these bootstrap results. I haven't read the paper closely (and would probably have made this a comment rather than an answer if I were able to), but it looks as though they argue that they can get consistent estimates with their method.

I would try BIC and compare the results with AIC. In my experience, BIC yields better and smaller models than AIC.

If you are using R, try these:

install.packages("BMA")
library(BMA)


In BIC, the Bayesian approach yields top 5-10 models that explains the data set. It allows you to investigate the models individually. Models are rated by posterior probability indicating their goodness of fit etc.

• In my experience BIC yields smaller and worse models if you use a proper accuracy score to judge. – Frank Harrell Sep 9 '15 at 12:38