3
$\begingroup$

I want to select a model which best performs for a very huge data set. However, the data set is too large to calculate a model within reasonable time.

If this is the case, is the following a reasonable approach: Fit each model to $n$ smaller random subsets of the original data set and calculate the mean AIC. Then, select the model with the lowest mean AIC.

$\endgroup$
5
  • 1
    $\begingroup$ To be clear; are you fitting the same model to every subset? $\endgroup$ – Stijn Sep 6 '14 at 8:45
  • $\begingroup$ Yes. I fit the same model to each subset. However, does this matter if $n$ is large? $\endgroup$ – Funkwecker Sep 8 '14 at 5:33
  • $\begingroup$ I imagine it does as different models will give different AIC values, and I don't know about the behaviour of the mean AIC value. $\endgroup$ – Stijn Sep 8 '14 at 7:17
  • 1
    $\begingroup$ What makes you need to select a model vs. fit a pre-specified model, ideally with penalization (shrinkage)? Note that the use of AIC for model selection is very similar to the highly disrespected stepwise regression. $\endgroup$ – Frank Harrell Sep 9 '15 at 12:39
  • $\begingroup$ Can you elaborate on why AIC for model selection is disrespected? Thx. $\endgroup$ – Funkwecker Apr 7 '16 at 13:26
1
$\begingroup$

What you're trying to do sounds a lot like what the authors of this paper propose: http://arxiv.org/abs/1112.5016

Essentially what they do is bootstrapping on small subsets of their very large dataset and then average over these bootstrap results. I haven't read the paper closely (and would probably have made this a comment rather than an answer if I were able to), but it looks as though they argue that they can get consistent estimates with their method.

$\endgroup$
0
$\begingroup$

I would try BIC and compare the results with AIC. In my experience, BIC yields better and smaller models than AIC.

If you are using R, try these:

install.packages("BMA")    
library(BMA)

In BIC, the Bayesian approach yields top 5-10 models that explains the data set. It allows you to investigate the models individually. Models are rated by posterior probability indicating their goodness of fit etc.

$\endgroup$
1
  • 1
    $\begingroup$ In my experience BIC yields smaller and worse models if you use a proper accuracy score to judge. $\endgroup$ – Frank Harrell Sep 9 '15 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.