12
$\begingroup$

I'm performing a Poisson Regression model with 1 response variable and 6 covariates. Model selection using AIC results in a model with all covariates as well as 6 interaction terms. The BIC however, results in a model with only 2 covariates and no interaction terms. Is it possible that the two criteria, that look very similar, yield totally different model selections?

$\endgroup$
  • 6
    $\begingroup$ If it wasn't possible to get different model recommendations from the two metrics, then there wouldn't be two metrics, we'd just always use one. $\endgroup$ – Gregor Thomas May 14 '18 at 23:59
  • $\begingroup$ The word "totally different" is hard to interpret when models are the result of choosing from a set of discrete parameters. $\endgroup$ – BallpointBen May 15 '18 at 3:20
21
$\begingroup$

It is possible indeed. As explained at https://methodology.psu.edu/AIC-vs-BIC, "BIC penalizes model complexity more heavily. The only way they should disagree is when AIC chooses a larger model than BIC."

If your goal is to identify a good predictive model, you should use the AIC. If your goal is to identify a good explanatory model, you should use the BIC. Rob Hyndman nicely summarizes this recommendation at
https://robjhyndman.com/hyndsight/to-explain-or-predict/:

"The AIC is better suited to model selection for prediction as it is asymptotically equivalent to leave-one-out cross-validation in regression, or one-step-cross-validation in time series. On the other hand, it might be argued that the BIC is better suited to model selection for explanation, as it is consistent."

The recommendation comes from Galit Shmueli’s paper “To explain or to predict?”, Statistical Science, 25(3), 289-310 (https://projecteuclid.org/euclid.ss/1294167961).

Addendum:

There is a third type of modeling - descriptive modeling - but I don't know of any references on which of AIC or BIC is best suited for identifying an optimal descriptive model. I hope others here can chime in with their insights.

| cite | improve this answer | |
$\endgroup$
  • 6
    $\begingroup$ The only way they should disagree is when AIC chooses a larger model than BIC.” Technically BIC could choose a larger model if $\ln n < 2$, i.e. $n \le 7$. Hopefully samples of size 7 aren’t too much of an issue, though. :p $\endgroup$ – Dougal May 15 '18 at 4:01
  • $\begingroup$ Good point! With a sample size of 7 or less, I would imagine model selection is off the table. 😀 $\endgroup$ – Isabella Ghement May 15 '18 at 20:37
  • $\begingroup$ There is a third type of modeling - descriptive modeling - but I don't know of any references on which of AIC or BIC is best suited for identifying an optimal descriptive model. I hope others here can chime in with their insights. Is it an answer or a question ? $\endgroup$ – Subhash C. Davar May 25 '18 at 8:41
  • $\begingroup$ @subhashc.davar: No answer yet - I'm tempted to email Galit Shmueli and ask her for her thoughts on that. $\endgroup$ – Isabella Ghement May 25 '18 at 14:57
  • 1
    $\begingroup$ If we understand the meaning of "descriptive" & take it seriously, I'm not sure it makes sense to talk about identifying the optimal descriptive model. $\endgroup$ – gung - Reinstate Monica May 25 '18 at 15:56
11
$\begingroup$

Short answer: yes, it is very possible. The two apply different penalties based on the number of estimated parameters (2k for AIC vs ln(n) x k for BIC, where k is the number of estimated parameters and n is the sample size). Thus, if the likelihood gain from adding a parameter is small, BIC may select different models to AIC. This effect is dependent on sample size, however.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ would be nice to make explicit that n is the sample size in the above equation $\endgroup$ – fabiob May 14 '18 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.