A fundamental property of random-effects regression is that the random intercept estimates are "shrunk" toward the overall mean of the response as a function of the relative variance of each estimate.

$$\hat{U}_j = \rho_j \bar{y}_j + (1-\rho_j)\bar{y}$$ where $$\rho_j = \tau^2 / (\tau^2 + \sigma^2/n_j).$$

This is also the case with generalized linear mixed models (GLMMs) such as logistic regression.

How is that shrinkage better than / different from fixed-effects logistic regression with one-hot-encoding of ID variables and shrinkage via L2-regularization?

In a fixed-effects model, I can control the amount of shrinkage by changing my penalty, $\lambda$, of L2-regularization while in a random-effects model I have no control on the amount of shrinkage. Would it be correct to say "use the random-effects model if the goal is inference but use the fixed-effects model if the goal is prediction"?


That's a bit oversimplified. The shrinkage in a mixed-effects regression is weighted by overall balance between "classes"/"groups" in the random-effects structures, so it's not that you don't get to to choose, but rather that your group size and strength of evidence chooses. (Think of it as like a weighted grand mean). Moreover, mixed-effects models are very useful when you have a number of groups but only very little data in each group: the overall structure and partial pooling allows for better inferences even within each group!

There are also LASSO (L1-regularized), ridge (L2-regularized), and elastic net (combination of L1 and L2 regularization) variants of mixed models. In other words, these things are orthogonal. In Bayesian terms, you get mixed-effects shrinkage via your hierarchical/multilevel model structure and regularization via your choice of prior on the distribution of model coefficients.

Perhaps the confusion arises from the frequent use of regularization in "machine learning" (where prediction is the goal) but the frequent use of mixed-effects in "statistics" (where inference is the goal), but that's more a side effect of other aspects of common datasets in such areas (e.g. size) and computational concerns. Mixed-effects models are generally harder to fit, so if a regularized fixed-effect model that ignores some structure of the data is good enough for the predictions you need, it may not be worthwhile to fit a mixed-effects model. But if you need to make inferences on your data, then ignoring its structure would be a bad idea.

  • $\begingroup$ precise and to the point answer. $\endgroup$ – Subhash C. Davar Mar 19 '17 at 14:23
  • $\begingroup$ Thanks Livius. Is the shrinkage of random effects similar to doing empirical Bayes? If yes, then would it still make sense to further shrink a random-effects model with L2-regularization/Bayesian on top? My goal is to rank the groups by BLUP and use the ranking in a next-stage prediction model. $\endgroup$ – Paman Gujral Mar 20 '17 at 17:48
  • $\begingroup$ I'm building a predictive model on a episode-level healthcare dataset that contains multiple hospital episodes per member ID. Most members have fewer than 5 episodes. I think this is a case where both a lasso or ridge regression applied to the fixed effects, plus a random effect for the member ID field, would be appropriate. $\endgroup$ – RobertF Aug 9 '17 at 21:11
  • 1
    $\begingroup$ @PamanGujral you may want to look at "Empirical Bayes Estimation of Random Effects Parameters in Mixed Effects Logistic Regression Models" by Ten Have and Localio $\endgroup$ – AdamO Apr 10 '18 at 18:13
  • $\begingroup$ "Mixed-effects models are generally harder to fit . . ." If the aim is to account for correlation between records sharing the same ID, and there are thousands or millions of unique IDs, adding a simple random intercept term to the regression formula using the formula listed in the OP's question seems like a reasonable and fairly simple first step. You're only estimating two parameters instead of a fixed effect term for every unique ID minus one, which saves far more degrees of freedom. $\endgroup$ – RobertF Aug 6 '18 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.