4
$\begingroup$

If I understand AIC/AICc vs. BIC correctly, AIC presumes that there is no "true" model and that any given model is simply a "best worst approximation". However, BIC presumes that there IS a "true" model and that the lowest BIC of a comparison represents the best approximation of the models present of that "true" model.

So, why do people use BIC for model averaging? Doesn't that contradict an important assumption behind BIC? Is it merely because BIC penalizes more harshly for additional terms? If so, does that mean the premise of a "true" model really doesn't matter?

Improve this question
$\endgroup$
7
  • $\begingroup$ A true model is needed for being able to talk about the consistency of BIC. However, absence of a true model does not make BIC useless. Intuitively, a model that approximates the true model relatively well should be ranked highly by BIC when compared to other models that are poorer approximations. $\endgroup$
    – Richard Hardy
    Commented Mar 15, 2016 at 15:53
  • $\begingroup$ That's not an answer to what I asked. I asked if BIC is theoretically incompatible with model averaging, which is not the same thing as model selection. I didn't ask whether or not BIC was consistent at all. $\endgroup$
    – Bryan
    Commented Mar 15, 2016 at 17:18
  • $\begingroup$ You said BIC presumes that there is a "true" model -- and I offered a case when this is relevant; it is relevant for the consistency of BIC. However, should this be relevant for other properties or uses of BIC such as model averaging? Perhaps not. At least the existence of a true model is not required for BIC to be well defined, or is it? $\endgroup$
    – Richard Hardy
    Commented Mar 15, 2016 at 17:27
  • $\begingroup$ That is still not any answer to what I asked. I will quote myself, since there seems to be some difficulty in understanding: "why do people use BIC for model averaging? Doesn't that contradict an important assumption behind BIC?" $\endgroup$
    – Bryan
    Commented Mar 15, 2016 at 17:30
  • 1
    $\begingroup$ First, I am commenting, not answering. I hope my comments could shed some light on the topic and be relevant (perhaps indirectly) to the question. Second, I have said two times already that the existence of a true model may not be an assumption behind BIC. Are you sure it is? (Do you have a reference?) If it is not, than your question should be reformulated as it contains a false statement. $\endgroup$
    – Richard Hardy
    Commented Mar 15, 2016 at 17:33

1 Answer 1

Reset to default
8
$\begingroup$

BIC assumes a true model, but you don't know what that model is.

Let's take a step back (and assume we're being Bayesians while we're at it).

Imagine we're trying to forecast where the true model is among a collection of possible models (the situation the BIC is for). We don't know which model is correct, so we can only update our prior probabilities on the models to posterior probabilities (given the data).

So imagine we have 48% posterior probability on one model, 39% posterior probability on a second model, and 12% on a third (with all other models in the collection sharing the remainder which I'll mostly ignore for now).

We want to make a forecast (e.g. give a 95% posterior prediction interval) that incorporates our uncertainty about which model is correct. Note that if we (quite incorrectly!) treat the model with the highest posterior probability as if it were known to be the model, we'd be asserting a level of certainty about our forecast interval we don't possess; it's conditional on a model being the correct one that we don't have good reason to say is the correct one. Indeed we are slightly more certain that particular model isn't the right model than we are that it is (our posterior probability that the first model is not the right model is 52%).

It would be reckless to ignore that uncertainty about which model is the correct one -- our forecast interval would be too narrow (often far too narrow).

Now BIC is (up to a common shift) minus twice an asymptotic approximation to the posterior probability (under the assumption of equal prior weight). So model averaging weights derived from BIC in the usual way correspond to posterior probabilities under a particular set of assumptions (including that there is one correct model). The same argument I gave above applies -- we don't know which one it is, and it would be reckless (indeed, quite wrong) to pretend we did.

It's not much different to saying "I know the population mean has one particular value" and then complaining when people want to give an interval for it (we don't know what they value is, so of course it's reasonable to entertain more than one possible value in our inference).

Improve this answer
$\endgroup$
13
  • $\begingroup$ We had some discussion with the OP on how important it is to have the true model within the set of candidate models. So my extra question is, how bad is it if the true model is not in the set of candidate models? What property of BIC-weighting breaks then? Do BIC-weight become useless? If not, what can we still claim? $\endgroup$
    – Richard Hardy
    Commented Mar 16, 2016 at 19:17
  • $\begingroup$ @RIchard It's a big question if we don't constrain it -- even as a new question I think it's too broad. As far as I can see, nothing prevents even the best members of your candidate model class from being arbitrarily bad at prediction if they don't contain the true model. It may be possible to say something about bounds that relate to some suitable distance measure between the true model and the nearest one in the class. I may have seen something like that but off the top of my head don't recall any such results. Alternatively, one could take a simpler tack and present some examples; ... ctd $\endgroup$
    – Glen_b
    Commented Mar 16, 2016 at 23:12
  • $\begingroup$ ctd ... but results would be hard to generalize -- one thing to do would be to try to see just how bad it could get. $\endgroup$
    – Glen_b
    Commented Mar 16, 2016 at 23:14
  • $\begingroup$ thank you for the explanation. Since having a true model within the candidate model set may have zero probability in many applied cases, the picture does not look bright (at least in theory). But I suppose in practice we do not worry about that too much. $\endgroup$
    – Richard Hardy
    Commented Mar 17, 2016 at 6:15
  • $\begingroup$ @Richard It may not be so bad. If the class of models is a sufficiently flexible one, it may include models that can come arbitrarily close to the true one ... at the expense, perhaps, of being more flexible than really needed (larger variances with nonparametric regression when trying to fit a linear relationship for example). That doesn't cover all possibilities (such as missing important predictors, for example), but it does at least suggest there's not necessarily reason to worry over-much about functional-form. $\endgroup$
    – Glen_b
    Commented Mar 17, 2016 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.