# AIC model averaging when models are correlated

AIC model-averaging: In "standard" AIC model averaging we average models with weights proportional to $$w_i \propto \exp( -0.5 \times \Delta \text{AIC}_i ),$$ where $$\Delta \text{AIC}_i$$ is the difference of a models AIC to the best (in terms of AIC) model.

What I noticed is that when some models are heavily correlated, this seems to not work so well in terms of prediction error on new unseen data.

Example: Let's take an exaggerated extreme case. Let's say we have model 1 and model 2 and both have the same AIC. So, $$w_1 = 0.5$$ and $$w_2 = 0.5$$. Now, we introduce additional models 1a, 1b and 1c that are effectively the same as (or extremely similar to) model 1. If we blindly apply the formula above, we end up with $$w_{1}=0.2$$, $$w_{1a}=0.2$$, $$w_{1b}=0.2$$, $$w_{1c}=0.2$$ and $$w_{2}=0.2$$. However, what we really ought to be doing is $$w_{1}=0.125$$, $$w_{1a}=0.125$$, $$w_{1b}=0.125$$, $$w_{1c}=0.125$$ and $$w_{2}=0.5$$.

Question: Do you know some simple results that e.g. look at the correlation of predictions from the model (or some other considerations) to take the "similarity" of the models into account when deciding model averaging weights?

In case it matters, I'm primarily asking in the context of models for prediction. I.e. I do not really care about selecting a single true model or determining the "independent" effect of some covariate, but primarily want good predictive performance on new unseen data from the same data generating mechanism.

My ideas/investigations: I've failed to find any literature that discusses this for AIC model averaging, it seems like a rather obvious questions, so I've probably missed something.

One thing I've thought of is to do k-fold cross-validation and doing non-negative regression on out-of-fold predictions to determine model weights, but that gets a whole lot more complicated than AIC model averaging. Thus, I'm interested in whether there's any work on this topic I've missed.

• Out of interest, do the two different weight assignments you mention lead to different (weighted) predictions?
– Eoin
Nov 2, 2020 at 20:04
• Yes, the highly correlated models get too much weight and influence the overall mean too much, while they ought to have less influence. Nov 2, 2020 at 23:52
• We touch on this in esajournals.onlinelibrary.wiley.com/doi/full/10.1002/ecm.1309 Mar 30, 2021 at 17:30