Why is it valid to account for k-1 intercepts w/ only 1 random intercept parameter?

I possess a basic understanding of random vs. fixed effects, and how to code random effects models in SAS. However, I'm having trouble wrapping my head around the derivation of random effects terms, and how a random intercept model, for example, can describe the variation in $k$ intercepts with a single parameter ($\sigma^2$ for a normal dbn) rather than $k-1$ parameters, which can amount to a huge saving in degrees of freedom. Isn't that cheating? ;)

The old school technique would be to use the maximum likelihood method to solve for $k-1$ parameters for each category.

• How does a random intercept model avoid this by using a single parameter?
• Isn't the end result the same--both random and fixed effect models will estimate $k-1$ intercept terms?
• Welcome to the site, @RobertF. I tried to give a more informative title to this question based on my interpretation of it. Make sure you're OK w/ the new title, & change it if you want the emphasis to differ, but it's best to have the title summarize the main issue clearly & succinctly. – gung Aug 27 '12 at 17:38

Things are a little more complicated with mixed effects models (as you are realizing).

The estimated random effects from the normal model are not exactly the same as the fixed effects that you would compute if you calculated each of the individual intercepts. The fixed effect model assumes that all the groups have the same variance, but each has its own mean and the means are computed independently of each other (basically the mean/intercept for each group). In the mixed effects model the assumption that the random intercepts come from a normal distribution allows the methods to "borrow" information from all the groups in calculating each intercept so the individual intercepts tend to be "shrunken" towards the overall mean, an effect that is often referred to as "regression to the mean".

The estimated random effects are not really parameters, but estimates of random variables, so they don't cost the same number of degrees of freedom. But you are correct in being concerned that there should only be a cost of 1 degree of freedom, the true cost is probably somewhere between $1$ and $k-1$. Here is a post that gives some more detail and an additional reference.

Of course discussing the degrees of freedom assumes that the ratios follow an F distribution, which is itself questionable in mixed effects models.

• That's a good answer @Greg. I'd just add that, usually, in a mixed effect model the main interest is in the fixed effects. Of course, that raises the question of what effect bad estimates of the random effects has on the fixed effects. – Peter Flom Aug 27 '12 at 18:45

as ive studied the random vs fixed effecs i neither could get the thing where u "estimate" parameters without loosing df. but the big difference as already mentioned is that the estimating of RE is not really a estimation in the classical OLS sense. in my understand it is better to call it "prediction" since we estimate the outcome of a random variable rather then of a fixed parameter of the given universe.

in case you can read stuff in german i could recommend you a book if you like.

• Welcome to the site, @Druss2k. We want answers to be clear, informative, & to stand on their own. Would you mind editing this for English grammar & clarity? Also, why don't you list the title & author of the book you recommend, w/ some info about how it's helpful (& also the caveat that it's only in German)? – gung Aug 27 '12 at 20:12
• Thank you @Druss2k, unfortunately I don't read German. Maybe there's a translation available? – RobertF Aug 28 '12 at 20:27
• Sry, I just recently remembered that I posted here.... Are you still interested in an answer? – Druss2k Dec 7 '12 at 15:55