3
$\begingroup$

I want to select a model which best performs for a very huge data set. However, the data set is too large to calculate a model within reasonable time.

If this is the case, is the following a reasonable approach: Fit each model to $n$ smaller random subsets of the original data set and calculate the mean AIC. Then, select the model with the lowest mean AIC.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ To be clear; are you fitting the same model to every subset? $\endgroup$
    – Stijn
    Sep 6, 2014 at 8:45
  • $\begingroup$ Yes. I fit the same model to each subset. However, does this matter if $n$ is large? $\endgroup$
    – Funkwecker
    Sep 8, 2014 at 5:33
  • $\begingroup$ I imagine it does as different models will give different AIC values, and I don't know about the behaviour of the mean AIC value. $\endgroup$
    – Stijn
    Sep 8, 2014 at 7:17
  • 1
    $\begingroup$ What makes you need to select a model vs. fit a pre-specified model, ideally with penalization (shrinkage)? Note that the use of AIC for model selection is very similar to the highly disrespected stepwise regression. $\endgroup$
    – Frank Harrell
    Sep 9, 2015 at 12:39
  • $\begingroup$ Can you elaborate on why AIC for model selection is disrespected? Thx. $\endgroup$
    – Funkwecker
    Apr 7, 2016 at 13:26

2 Answers 2

Reset to default
1
$\begingroup$

What you're trying to do sounds a lot like what the authors of this paper propose: http://arxiv.org/abs/1112.5016

Essentially what they do is bootstrapping on small subsets of their very large dataset and then average over these bootstrap results. I haven't read the paper closely (and would probably have made this a comment rather than an answer if I were able to), but it looks as though they argue that they can get consistent estimates with their method.

Improve this answer
$\endgroup$
0
$\begingroup$

I would try BIC and compare the results with AIC. In my experience, BIC yields better and smaller models than AIC.

If you are using R, try these:

install.packages("BMA")    
library(BMA)

In BIC, the Bayesian approach yields top 5-10 models that explains the data set. It allows you to investigate the models individually. Models are rated by posterior probability indicating their goodness of fit etc.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ In my experience BIC yields smaller and worse models if you use a proper accuracy score to judge. $\endgroup$
    – Frank Harrell
    Sep 9, 2015 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.