10
$\begingroup$

A fundamental property of random-effects regression is that the random intercept estimates are "shrunk" toward the overall mean of the response as a function of the relative variance of each estimate.

$$\hat{U}_j = \rho_j \bar{y}_j + (1-\rho_j)\bar{y}$$ where $$\rho_j = \tau^2 / (\tau^2 + \sigma^2/n_j).$$

This is also the case with generalized linear mixed models (GLMMs) such as logistic regression.

How is that shrinkage better than / different from fixed-effects logistic regression with one-hot-encoding of ID variables and shrinkage via L2-regularization?

In a fixed-effects model, I can control the amount of shrinkage by changing my penalty, $\lambda$, of L2-regularization while in a random-effects model I have no control on the amount of shrinkage. Would it be correct to say "use the random-effects model if the goal is inference but use the fixed-effects model if the goal is prediction"?

$\endgroup$

1 Answer 1

13
$\begingroup$

That's a bit oversimplified. The shrinkage in a mixed-effects regression is weighted by overall balance between "classes"/"groups" in the random-effects structures, so it's not that you don't get to to choose, but rather that your group size and strength of evidence chooses. (Think of it as like a weighted grand mean). Moreover, mixed-effects models are very useful when you have a number of groups but only very little data in each group: the overall structure and partial pooling allows for better inferences even within each group!

There are also LASSO (L1-regularized), ridge (L2-regularized), and elastic net (combination of L1 and L2 regularization) variants of mixed models. In other words, these things are orthogonal. In Bayesian terms, you get mixed-effects shrinkage via your hierarchical/multilevel model structure and regularization via your choice of prior on the distribution of model coefficients.

Perhaps the confusion arises from the frequent use of regularization in "machine learning" (where prediction is the goal) but the frequent use of mixed-effects in "statistics" (where inference is the goal), but that's more a side effect of other aspects of common datasets in such areas (e.g. size) and computational concerns. Mixed-effects models are generally harder to fit, so if a regularized fixed-effect model that ignores some structure of the data is good enough for the predictions you need, it may not be worthwhile to fit a mixed-effects model. But if you need to make inferences on your data, then ignoring its structure would be a bad idea.

$\endgroup$
5
  • $\begingroup$ precise and to the point answer. $\endgroup$
    – user10619
    Mar 19, 2017 at 14:23
  • $\begingroup$ Thanks Livius. Is the shrinkage of random effects similar to doing empirical Bayes? If yes, then would it still make sense to further shrink a random-effects model with L2-regularization/Bayesian on top? My goal is to rank the groups by BLUP and use the ranking in a next-stage prediction model. $\endgroup$ Mar 20, 2017 at 17:48
  • $\begingroup$ I'm building a predictive model on a episode-level healthcare dataset that contains multiple hospital episodes per member ID. Most members have fewer than 5 episodes. I think this is a case where both a lasso or ridge regression applied to the fixed effects, plus a random effect for the member ID field, would be appropriate. $\endgroup$
    – RobertF
    Aug 9, 2017 at 21:11
  • 1
    $\begingroup$ @PamanGujral you may want to look at "Empirical Bayes Estimation of Random Effects Parameters in Mixed Effects Logistic Regression Models" by Ten Have and Localio $\endgroup$
    – AdamO
    Apr 10, 2018 at 18:13
  • $\begingroup$ "Mixed-effects models are generally harder to fit . . ." If the aim is to account for correlation between records sharing the same ID, and there are thousands or millions of unique IDs, adding a simple random intercept term to the regression formula using the formula listed in the OP's question seems like a reasonable and fairly simple first step. You're only estimating two parameters instead of a fixed effect term for every unique ID minus one, which saves far more degrees of freedom. $\endgroup$
    – RobertF
    Aug 6, 2018 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.