Regularization via model averaging?

Question

Say you have the model $$ \Phi^{-1} \left(y\right)=\beta_0 + \beta_1 x$$ I am interested in adding some regularization, specifically concerning the parameter $\beta_1$, to introduce some "skepticism" toward large values.

I know I could use penalized likelihood methods (e.g. LASSO or Ridge), but I came up with this other approach, which also would avoid the need of selecting the value of the regularization coefficient.

The alternative approach would proceed in this way:

1. Estimate a reduced model (equivalent to fixing $\beta_1=0$) $$ \Phi^{-1} \left(y\right)=\beta_0 $$
2. Calculate the AIC: $$\text{AIC}_1 = 4 - 2 \ln \mathcal{L}_1$$ $$\text{AIC}_0 = 2 - 2 \ln \mathcal{L}_0$$ where $\mathcal{L}_1$ and $\mathcal{L}_0$ are the maximized likelihoods of the full and reduced models, respectively.
3. Transform them in differences with respect to the best model ${\Delta _m} = {\rm{AI}}{{\rm{C}}_m} - \min {\rm{AIC}}$ and calculate the Akaike weights $${a_m} = \frac{{\exp \left( { - \frac{1}{2}{\Delta_m}} \right)}}{{\mathop \sum \limits_{j } \exp \left( { - \frac{1}{2}{\Delta_j}}\right)}}$$
4. Calculate the model-averaged estimate $$\bar \beta_1 = a_0 \beta_{1,0} + a_1 \beta_{1,1} = a_1 \beta_{1,1}$$ (since $\beta_{1,0}=0$ by construction in the reduced model). Following Buckland et al. (1997) the standard error of the model averaged parameter would be $$ \begin{align} \hat \sigma_{\bar \beta_1} & = \sum\limits_{j} {a_j} \sqrt{ \sigma_{\beta_j}^2 + \left(\beta_j - \bar \beta \right)^2 } \\ & = a_0 \beta_{1,0} + a_1 \sqrt{ \sigma_{\beta_{1,1}}^2 + \left(\beta_{1,1} - \bar \beta_1 \right)^2 } \end{align} $$ (again because in the reduced model $\beta_{1,0}=0$ and $\sigma_{\beta_{1,0}}=0$)

This procedure thus should result in a estimate of the coefficient $\beta_1$ that is shrunk toward zero by an amount that depends on the difference in AIC between the full and reduced model: the less useful is $x$ for predicting $y$, the more the estimate of $\beta_1$ will shrink toward zero.

It seems correct to me to say that AIC could be interpreted as a type of regularization (e.g. this question) but I was wondering if this approach of using AIC averaging for the specific purpose of regularizing the estimated value of a single parameter has been described before? Is there any relevant reference? In which cases LASSO or Ridge penalty should be preferred, and in which cases instead this approach could be useful?

Answer 1

Yes it is a valid approach in the sense that it's been proposed and used in the published literature:

See for example this review paper on model selection and model averaging where the specific technique you propose is discussed on pages 16 (for the weights) and 18 for the single parameter estimation based on AIC weights:

http://byrneslab.net/classes/biol607/readings/Symonds_and_Moussalli_2010_behav_ecol.pdf

They also trace this approach of estimating a parameter based on AIC model averaging back to it's origin: Specifically within the ecology literature this AIC based model averaging approach traces back to this paper (which also proposes a couple of other model selection approaches): https://pdfs.semanticscholar.org/0c88/444cf4be49515b9d80b263b5fbeb62866899.pdf

I was not able to find/don't know of a reference comparing this approach to say a LASSO or ridge regression for parameter estimation.

The rest of this answer is probably not relevant to using it to estimate a single parameter but rather in using this approach for model building:

The authors also discuss some issues in this approach (AIC based model averaging) though these are pretty much common to all model selection approaches: if you have a bunch of models with essentially meaningless variables then this approach will still provide a "best" model. So it is important to not just use this procedure in automation but as part of a larger approach that also includes seeing how well the model (especially the "best" model fits the data). This is a fundamental issue in model selection techniques (of which model averaging is one approach): the "best" model in scientific terms is often different from the "best" model in statistical terms (e.g., just considering the fitting of model to data).

So with that caveat aside the technique you proposed is valid though there are questions about if it is the best technique. There are some specific concerns about whether or not it actually improves on the statistical performance:

https://esajournals.onlinelibrary.wiley.com/doi/abs/10.1890/05-0074 <- shows that depending on the scenario investigated (they consider two ecological problems where they generate the data from a known model and then ask how techniques such as AIC model averaging compare to single model approaches). In one scenario model averaging improved the predictions and in another it didn't.

https://link.springer.com/article/10.1007/s00265-010-1035-8 <- employs a similar approach as the previous reference to show examples where AIC model averaging works and doesn't work (in terms of predictive accuracy).

Though https://link.springer.com/article/10.1007/s10463-009-0234-4 shows that it performs better than stepwise selection techniques and that AIC based model averaging can provide nominal coverage for confidence intervals.

So as a whole there is a lot to dive into and whether or not this particular techniques works better than other model selection techniques is probably dependent on the exact problem, the complexity of the models, the availability of data, etc.