I am trying to get my head around exactly how mixed-effect models produce the estimates they produce. Say you have 20 students from 100 classrooms each (200 students altogether) and you're trying to regress minutes spent studying onto GPA. My understanding of the general concept is that the model will estimate "fixed-effects," that is, what the intercept and slope would be if you ran one big regression and ignored the clustering by classroom. Then, it produces "random-effects" that are essentially classroom-specific deviations for the intercept and slope parameters. What I don't understand is how these 40 estimates (20 random intercepts, 20 random slopes) are produced. If they are just assumed to be drawn from a random normal with mean 0, how does that improve anything? Take classroom 1. Compared to classrooms 2 through 39, it has an above-average intercept and a below-average slope. But if the random-effects (deviations from grand-intercept and grand-slope) are both drawn from normals with mean 0, what's preventing the random effects from estimating a below-average intercept and an above-average slope? How does the model fitting find some "medium-ground" (partial pooling) between the grand-intercept/slope and the intercept/slope it would produce if it just ran 20 independent regressions? How does the shrinkage produced by the added mean 0 noise aid inference?


1 Answer 1


There are several issues here. First I try to be precise about terminology. The random effect for classroom has only one parameter, namely its variance. Its mean is fixed at zero. Only one parameter is estimated here in the standard statistical sense, although one can, based on this estimate, also predict the specific values taken by the random effect. This is not called "estimation" but "prediction" because these are not model parameters but rather realisations of random variables that are unobserved.

The mean zero assumption for the random effects is an identifiability assumption, meaning that if the mean were something else, you'd have an equivalent model if you'd also change the fixed overall intercept and slope, the parameters would not be identified and could not be estimated. This is not different from if your random effects were modelled as fixed effects. You'd then need an identifiability assumption of the kind that they all sum up to zero (sometimes other identifiability assumptions are used that change the interpretation of the parameters but don't change the overall model).

What is won by using a random effects model is in the first place a reduction of the number of parameters to estimate, leading to more stable estimators and to what some people call "borrowing strength", meaning that if you don't have much data/information for a single effect value (e.g. specific to one classroom), overall distribution information over all values (classrooms) can be used to say something more specific about the one about which you'd otherwise observe very little.

This is good but also to some extent risky, as the "prediction" for this (random) effect relies more on information "elsewhere in the data" than if it were a fixed effect for the specific classroom, and it is usually hard if not impossible to check to what extent this "borrowing" of information is reliable. This is a reason why, in a situation in which (a) you have enough data for every single classroom and (b) you want reliable prediction within that classroom, a fixed effect would usually be preferable. A standard recommendation is that fixed effects are used if the focus of your interest are the specific effects for your factor levels, which may also be relevant for predicting future observations, whereas random effects are typically applied if in future you are interested in new factor levels (i.e., further classrooms, not the ones already in your study), for which a fixed effects model doesn't allow predictions.

The "shrinkage" of random-effect predictions is hard to explain in intuitive terms. What I can say is this:

  1. There are two sources of variation to be taken into account when predicting a random effect, namely the variation in the random effects themselves, and the variation at individual observation level (this is different from fitting 20 different regressions).

  2. Formally, shrinkage does not happen conditionally on the observations; the random effect is predicted as its expected value conditionally on the observations, and this looks like no shrinkage takes place. Shrinkage only happens looking at the unconditional distribution of the predictions.

  3. The way I see it, this shrinkage happens because the assumption that the random effects distribution is normal with mean zero internally translates to additional information that predictions "if in doubt" (i.e., if the predictions are based on not many observations at that factor level) should be chosen closer to zero than what they would seem without that assumption (this bias goes to zero if the number of observations at that factor level grows). If I understand things correctly, this could only be "repaired" at the cost of having a larger mean squared error. The requirement to balance the two different sources of variation renders the situation asymmetric in such a way that either one can choose random effect predictions on average slightly too close to zero, or one reduces their precision so that the expected squared error is ultimately larger, as predictions larger in absolute value come with a bigger potential for imprecision.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.