L2-regularization vs random effects shrinkage

Question

A fundamental property of random-effects regression is that the random intercept estimates are "shrunk" toward the overall mean of the response as a function of the relative variance of each estimate.

$$\hat{U}_j = \rho_j \bar{y}_j + (1-\rho_j)\bar{y}$$ where $$\rho_j = \tau^2 / (\tau^2 + \sigma^2/n_j).$$

This is also the case with generalized linear mixed models (GLMMs) such as logistic regression.

How is that shrinkage better than / different from fixed-effects logistic regression with one-hot-encoding of ID variables and shrinkage via L2-regularization?

In a fixed-effects model, I can control the amount of shrinkage by changing my penalty, $\lambda$, of L2-regularization while in a random-effects model I have no control on the amount of shrinkage. Would it be correct to say "use the random-effects model if the goal is inference but use the fixed-effects model if the goal is prediction"?

Answer 1

That's a bit oversimplified. The shrinkage in a mixed-effects regression is weighted by overall balance between "classes"/"groups" in the random-effects structures, so it's not that you don't get to to choose, but rather that your group size and strength of evidence chooses. (Think of it as like a weighted grand mean). Moreover, mixed-effects models are very useful when you have a number of groups but only very little data in each group: the overall structure and partial pooling allows for better inferences even within each group!

There are also LASSO (L1-regularized), ridge (L2-regularized), and elastic net (combination of L1 and L2 regularization) variants of mixed models. In other words, these things are orthogonal. In Bayesian terms, you get mixed-effects shrinkage via your hierarchical/multilevel model structure and regularization via your choice of prior on the distribution of model coefficients.

Perhaps the confusion arises from the frequent use of regularization in "machine learning" (where prediction is the goal) but the frequent use of mixed-effects in "statistics" (where inference is the goal), but that's more a side effect of other aspects of common datasets in such areas (e.g. size) and computational concerns. Mixed-effects models are generally harder to fit, so if a regularized fixed-effect model that ignores some structure of the data is good enough for the predictions you need, it may not be worthwhile to fit a mixed-effects model. But if you need to make inferences on your data, then ignoring its structure would be a bad idea.