9
$\begingroup$

AIC model-averaging: In "standard" AIC model averaging we average models with weights proportional to $$w_i \propto \exp( -0.5 \times \Delta \text{AIC}_i ),$$ where $\Delta \text{AIC}_i$ is the difference of a models AIC to the best (in terms of AIC) model.

What I noticed is that when some models are heavily correlated, this seems to not work so well in terms of prediction error on new unseen data.

Example: Let's take an exaggerated extreme case. Let's say we have model 1 and model 2 and both have the same AIC. So, $w_1 = 0.5$ and $w_2 = 0.5$. Now, we introduce additional models 1a, 1b and 1c that are effectively the same as (or extremely similar to) model 1. If we blindly apply the formula above, we end up with $w_{1}=0.2$, $w_{1a}=0.2$, $w_{1b}=0.2$, $w_{1c}=0.2$ and $w_{2}=0.2$. However, what we really ought to be doing is $w_{1}=0.125$, $w_{1a}=0.125$, $w_{1b}=0.125$, $w_{1c}=0.125$ and $w_{2}=0.5$.

Question: Do you know some simple results that e.g. look at the correlation of predictions from the model (or some other considerations) to take the "similarity" of the models into account when deciding model averaging weights?

In case it matters, I'm primarily asking in the context of models for prediction. I.e. I do not really care about selecting a single true model or determining the "independent" effect of some covariate, but primarily want good predictive performance on new unseen data from the same data generating mechanism.

My ideas/investigations: I've failed to find any literature that discusses this for AIC model averaging, it seems like a rather obvious questions, so I've probably missed something.

One thing I've thought of is to do k-fold cross-validation and doing non-negative regression on out-of-fold predictions to determine model weights, but that gets a whole lot more complicated than AIC model averaging. Thus, I'm interested in whether there's any work on this topic I've missed.

$\endgroup$
3
  • $\begingroup$ Out of interest, do the two different weight assignments you mention lead to different (weighted) predictions? $\endgroup$
    – Eoin
    Nov 2, 2020 at 20:04
  • $\begingroup$ Yes, the highly correlated models get too much weight and influence the overall mean too much, while they ought to have less influence. $\endgroup$
    – Björn
    Nov 2, 2020 at 23:52
  • 1
    $\begingroup$ We touch on this in esajournals.onlinelibrary.wiley.com/doi/full/10.1002/ecm.1309 $\endgroup$ Mar 30, 2021 at 17:30

1 Answer 1

5
+50
$\begingroup$

To the best of my knowledge, such a modification of the weights in Bayesian Model Averaging to take the similarity (or other relations) between models into account does not exist in the literature. According to me, the main reason is that the problem your are raising (and that you nicely illustrated in your example) should be corrected at the level of models selection, and not at the level of model averaging.

As far as I know, a characterization of the "similarity" of the models does not exist, and would anyway be difficult to define. Even a notion as simple and widely used as "nestedness" lacks a rigorous definition in the literature (reference) (although we proposed a definition in this recent paper). Different models might have the same prediction, while being greatly different in their structure and nature. If a phenomenological, a normative, and a physical models all agree on the same prediction, then the evidence for the said prediction is very high, and these models "deserve" to have an important weight in your model averaging (even if they have the same prediction).

In your example, the problem is from the choice of the models, not the averaging itself. The family of models 1, 1a, 1b, 1c, 2 is ill-defined : it is like sampling only a small part of your population (around model 1), which will lead to a biased result. However, apart from heuristically checking if your proposed family of models is sound, I don't think there exists (yet) a quantitative criterion or method to avoid this pitfall.

$\endgroup$
2
  • $\begingroup$ Re: "...corrected at the level of models selection, and not at the level of model averaging" - I guess there's the issue of model selection leading to problematic out-of-sample predictions, which can be helped with model averaging. You may end up with models that are somewhat similar and some that are less similar, the example I gave is of course ill-defined, but was more meant as an extreme case that illustrates (and is a case where we "know" the correct answer). $\endgroup$
    – Björn
    Nov 5, 2020 at 14:22
  • $\begingroup$ In part, I'm thinking of a use case where you only get aggregate information. Imagine a couple of scenarios: 1) someone trained a model and published it, but you only have the model and the reported AICs, but not the training data or 2) something like a Kaggle-style data science competition, where you get to see a aggregate log-loss on a dataset you do not have access to (or other similar settings), where you now want to average models on that basis (I think in the latter scenario you could ignore the number of parameters, but otherwise it's very similar). And there's probably more use cases. $\endgroup$
    – Björn
    Nov 5, 2020 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.