13
$\begingroup$

Is there an intuitive reason for random effects to be shrunk towards their expected value in the general linear mixed model?

$\endgroup$
3
  • $\begingroup$ Can you please provide some more context for this question? $\endgroup$
    – Macro
    Commented Jul 18, 2012 at 11:18
  • 1
    $\begingroup$ Predicted values from random-effect models are shrinkage estimators; there will be little skrinkage when statistical units are different, or when measurements are accurate, or with large sample. Is this what you are after, or do you really mean shrinkage toward expected value? $\endgroup$
    – chl
    Commented Jul 18, 2012 at 13:45
  • 4
    $\begingroup$ I would suggest an older article by Bradley Efron and Carl Morris, Stein's Paradox in Statistics (1977) (an online PDF is here). Not sure if it is intuitive, but it is a pretty gentle introduction (with real world examples) into the concept of shrinkage. $\endgroup$
    – Andy W
    Commented Jul 18, 2012 at 16:50

3 Answers 3

5
$\begingroup$

generally speaking, most "random effects" occur in situations where there is also a "fixed effect" or some other part of the model. The general linear mixed model looks like this:

$$y_i=x_i^T\beta+z_i^Tu+\epsilon_i$$

Where $\beta$ is the "fixed effects" and $u$ is the "random effects". Clearly, the distinction can only be at the conceptual level, or in the method of estimation of $u$ and $\beta$. For if I define a new "fixed effect" $\tilde{x}_i=(x_i^T,z_i^T)^T$ and $\tilde{\beta}=(\beta^T,u^T)^T$ then I have an ordinary linear regression:

$$y_i=\tilde{x}_i^T\tilde{\beta}+\epsilon_i$$

This is often a real practical problem when it comes to fitting mixed models when the underlying conceptual goals are not clear. I think the fact that the random effects $u$ are shrunk toward zero, and that the fixed effects $\beta$ are not provides some help here. This means that we will tend to favour the model with only $\beta$ included (i.e. $u=0$) when the estimates of $u$ have low precision in the OLS formulation, and tend to favour the full OLS formulation when the estimates $u$ have high precision.

$\endgroup$
0
2
$\begingroup$

Doesn't your question answer itself? If a value is expected then a technique that brings values closer to that would be best.

A simple answer comes from the law of large numbers. Let's say subjects are your random effect. If you run subjects A through D in 200 trials and subject E in 20 trials which of the subject's measured mean performance do you think is more representative of mu? The law of large numbers would predict that subject E's performance will be more likely to deviate by a larger amount from mu than any of A through D. It may or may not, and any of the subjects could deviate, but we would be much more justified in shrinking subject E's effect toward subject's A through D than the other way around. So random effects that are larger and have smaller N's tend to be the ones that are shrunk the most.

From this description also comes why fixed effects are not shrunk. It's because they're fixed,there's only one in the model. You have no reference to shrink it toward. You could use a slope of 0 as reference but that's not what random effects are shrunk toward. They're toward an overall estimate such as mu. The fixed effect that you have from your model is that estimate.

$\endgroup$
0
1
$\begingroup$

I think it might be helpful to your intuition to think of a mixed model as a hierarchical or multilevel model. At least to me, it makes more sense when I think of nesting and how the model is working within and across categories in a hierarchical manner.

EDIT: Macro, I had left this a little open-ended because it does help me view it more intuitively, but I'm not sure it's correct. But to expand it in possibly incorrect directions...

I look at it as fixed effects averaging across categories and random effects distinguishing between categories. In some sense, the random effects are "clusters" that share some characteristics, and larger and more compact clusters will have greater influence over the average at the higher level.

With OLS doing the fitting (in phases, I believe), larger and more compact random effect "clusters" will thus pull the fit more strongly towards themselves, while smaller or more diffused "clusters" will pull the fit less. Or perhaps the fit begins closer to larger and more compact "clusters" since the higher-level average is closer to begin with

Sorry I can't be clearer, and may even be wrong. It makes sense to me intuitively, but as I try to write it I'm not sure if it's a top-down or bottom-up thing, or something different. Is it a matter of lower-level "clusters" pulling fits towards themselves more strongly, or of having greater influence over the higher-level averaging -- and thus "ending up" nearer to the higher-level average -- or neither?

In either case, I feel that it explains why smaller, more diffuse categories of random variables will be pulled farther towards the mean than larger, more compact categories.

$\endgroup$
2
  • $\begingroup$ Hi Wayne, can you expand upon this to describe how the shrinkage can be (perhaps more intuitively) conceptualized by thinking of this as a hierarchical model? $\endgroup$
    – Macro
    Commented Jul 18, 2012 at 15:00
  • $\begingroup$ @Macro: OK, I gave it a try. Not sure if it makes the answer better or worse, though. $\endgroup$
    – Wayne
    Commented Jul 20, 2012 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.