AIC model averaging when models are correlated

Question

AIC model-averaging: In "standard" AIC model averaging we average models with weights proportional to $$w_i \propto \exp( -0.5 \times \Delta \text{AIC}_i ),$$ where $\Delta \text{AIC}_i$ is the difference of a models AIC to the best (in terms of AIC) model.

What I noticed is that when some models are heavily correlated, this seems to not work so well in terms of prediction error on new unseen data.

Example: Let's take an exaggerated extreme case. Let's say we have model 1 and model 2 and both have the same AIC. So, $w_1 = 0.5$ and $w_2 = 0.5$. Now, we introduce additional models 1a, 1b and 1c that are effectively the same as (or extremely similar to) model 1. If we blindly apply the formula above, we end up with $w_{1}=0.2$, $w_{1a}=0.2$, $w_{1b}=0.2$, $w_{1c}=0.2$ and $w_{2}=0.2$. However, what we really ought to be doing is $w_{1}=0.125$, $w_{1a}=0.125$, $w_{1b}=0.125$, $w_{1c}=0.125$ and $w_{2}=0.5$.

Question: Do you know some simple results that e.g. look at the correlation of predictions from the model (or some other considerations) to take the "similarity" of the models into account when deciding model averaging weights?

In case it matters, I'm primarily asking in the context of models for prediction. I.e. I do not really care about selecting a single true model or determining the "independent" effect of some covariate, but primarily want good predictive performance on new unseen data from the same data generating mechanism.

My ideas/investigations: I've failed to find any literature that discusses this for AIC model averaging, it seems like a rather obvious questions, so I've probably missed something.

One thing I've thought of is to do k-fold cross-validation and doing non-negative regression on out-of-fold predictions to determine model weights, but that gets a whole lot more complicated than AIC model averaging. Thus, I'm interested in whether there's any work on this topic I've missed.

Answer 1

To the best of my knowledge, such a modification of the weights in Bayesian Model Averaging to take the similarity (or other relations) between models into account does not exist in the literature. According to me, the main reason is that the problem your are raising (and that you nicely illustrated in your example) should be corrected at the level of models selection, and not at the level of model averaging.

As far as I know, a characterization of the "similarity" of the models does not exist, and would anyway be difficult to define. Even a notion as simple and widely used as "nestedness" lacks a rigorous definition in the literature (reference) (although we proposed a definition in this recent paper). Different models might have the same prediction, while being greatly different in their structure and nature. If a phenomenological, a normative, and a physical models all agree on the same prediction, then the evidence for the said prediction is very high, and these models "deserve" to have an important weight in your model averaging (even if they have the same prediction).

In your example, the problem is from the choice of the models, not the averaging itself. The family of models 1, 1a, 1b, 1c, 2 is ill-defined : it is like sampling only a small part of your population (around model 1), which will lead to a biased result. However, apart from heuristically checking if your proposed family of models is sound, I don't think there exists (yet) a quantitative criterion or method to avoid this pitfall.