I have found a lot on the internet regarding the interpretation of random- and fixed-effects. However I could not get a source pinning down the following:

What is the mathematical difference between random- and fixed-effects?

By that I mean the mathematical formulation of the model and the way parameters are estimated.

  • 1
    $\begingroup$ Well, fixed effects effect the mean of a joint distribution and random effects effect the variance and association structure. What exactly do you mean by the "mathematical difference"? Are you asking how the the likelihood changes? Can you be more specific? $\endgroup$ – Macro Apr 10 '12 at 22:12
  • $\begingroup$ Of possible interest: What is the difference between random effects-, fixed effects- & marginal model? $\endgroup$ – gung - Reinstate Monica Nov 24 '14 at 20:26
  • 1
    $\begingroup$ Also related: What is the difference between fixed effect, random effect and mixed effect models? $\endgroup$ – amoeba May 4 '16 at 21:26
  • 1
    $\begingroup$ The question does not seem to distinguish the background from which it is being drawn. This terminology in Panel Data Economics is different from the one in other social sciences using Multilevel Models. The question requires further clarification. Else, this is misleading for those arriving here from either background not knowing that there is an alternative definition in a related field. $\endgroup$ – luchonacho Jul 18 '16 at 12:14

The simplest model with random effects is the one-way ANOVA model with random effects, given by observations $y_{ij}$ with distributional assumptions: $$(y_{ij} \mid \mu_i) \sim_{\text{iid}} {\cal N}(\mu_i, \sigma^2_w), \quad j=1,\ldots,J, \qquad \mu_i \sim_{\text{iid}} {\cal N}(\mu, \sigma^2_b), \quad i=1,\ldots,I.$$

Here the random effects are the $\mu_i$. They are random variables, whereas they are fixed numbers in the ANOVA model with fixed effects.

For example each of three technicians $i=1,2,3$ in a laboratory records a series of measurements, and $y_{ij}$ is the $j$-th measurement of technician $i$. Call $\mu_i$ the "true mean value" of the series generated by technician $i$; this is a slightly artificial parameter, you can see $\mu_i$ as the mean value that technician $i$ would have been obtained if he/she had recorded a huge series of measurements.

If you are interested in evaluating $\mu_1$, $\mu_2$, $\mu_3$ (for example in order to assess the bias between operators), then you have to use the ANOVA model with fixed effects.

You have to use the ANOVA model with random effects when you are interested in the variances $\sigma^2_w$ and $\sigma^2_b$ defining the model, and the total variance $\sigma^2_b+\sigma^2_w$ (see below). The variance $\sigma^2_w$ is the variance of the recordings generated by one technician (it is assumed to be the same for all technicians), and $\sigma^2_b$ is called the between-technicians variance. Maybe ideally, the technicians should be selected at random.

This model reflects the decomposition of variance formula for a data sample : enter image description here

Total variance = variance of means $+$ means of intra-variances

which is reflected by the ANOVA model with random effects: enter image description here

Indeed, the distribution of $y_{ij}$ is defined by its conditional distribution $(y_{ij})$ given $\mu_i$ and by the distribution of $\mu_i$. If one computes the "unconditional" distribution of $y_{ij}$ then we find $\boxed{y_{ij} \sim {\cal N}(\mu, \sigma^2_b+\sigma^2_w)}$.

See slide 24 and slide 25 here for better pictures (you have to save the pdf file to appreciate the overlays, don't watch the online version).

  • 1
    $\begingroup$ (+1) Very nice figures! $\endgroup$ – amoeba Feb 26 '14 at 23:24
  • 1
    $\begingroup$ Thank you @amoeba, my code for the inertia moments is availabe on my blog: stla.github.io/stlapblog/posts/Variance_inertia.html $\endgroup$ – Stéphane Laurent Feb 27 '14 at 7:05
  • $\begingroup$ I don't get it. If I have a number of measurements performed by a number of technicians, why do I need an ANOVA? Can I not just fit a gaussian to each technician's results, and get a $\mu$ and $\sigma$ for each of them? What does your way of solving this allow me to do, which my way does not? $\endgroup$ – TheChymera Jan 15 '16 at 9:58
  • $\begingroup$ @TheChymera ANOVA is the assumption of a common $\sigma$. You get shorter confidence intervalw with this assumption. But your comment is about the reasons to use an ANOVA with common variance vs an ANOVA with different variances, this is not really the topic here. $\endgroup$ – Stéphane Laurent Jan 15 '16 at 16:10
  • $\begingroup$ @StéphaneLaurent Which ANOVA is the assumption of a common $\sigma$? - also, what things is this $\sigma$ common to? You said "If you are interested in evaluating μ1, μ2, μ3 (for example in order to assess the bias between operators), then you have to use the ANOVA model with fixed effects." What is the formula of the ANOVA method with fixed effects, and how does it inform you on $\mu_i$ without informing you on $\sigma_b^2$? Also, how can it give you an estimate of $\mu_i$ without providing all the required info to calculate of $\sigma_w^2$? (and vice-versa for the random effects model) $\endgroup$ – TheChymera Jan 15 '16 at 16:31

Basically, what I think is the most distinct difference if you model a factor as random, is that the effects are assumed to be drawn from a common normal distribution.

For example, if you have some sort of model regarding grades and you want to account for your student data coming from different schools and you model school as a random factor this means that you assume that the by school averages are normally distributed. That means two sources of variation are modelling: the in-school variability of student grades and the between school variability.

This results in something called partial pooling. Consider two extremes:

  1. School does not have any effect (between school variability is zero). In this case a linear model which does not account for school would be optimal.
  2. School variability is larger than student variability. Then you basically need to work on the school level instead of the students level (less # samples). This is basically the model where you account for school using fixed effects. This can be problematic if you have few samples per school.

By estimating the variability at both levels the mixed model makes a smart compromise between these two approaches. Especially if you have a not so large #students per school this means that you will get shrinkage of the effects for the individual schools as estimated by model 2 towards the overall mean of model 1.

That is because the models says that if you have one school with two students included which is better than what is "normal" for the population of schools then it is likely that part of this effect is explained by the school having been lucky in the choice of the two students looked at. It does not make this blindly, it does so depending on the estimate of the within school variability. This also means that effect levels with fewer samples are more strongly pulled toward the overall mean than large schools.

The important thing is that you need exchangeability on the levels of the random factor. That means in this case that the schools are (from your knowledge) exchangeable and you know nothing which makes them distinct (other than some sort of ID). If you have additional information you can include this as an additional factor, it is enough that the schools are exchangeable conditional on the other information accounted for.

For example, it would make sense to assume that 30 year old adults living in New York are exchangeable conditional on gender. If you have more information (age, ethnicity, education) it would make sense to include that information as well.

OTH if you have study with one control group and three wildly different disease groups it does not make sense to model group as random since specific disease are not exchangeable. However, many people like the shrinkage effect so well that they would still argue for a random effects model but that's another story.

I notice I didn't get too much into the mathematics, but basically the difference is that the random effects model estimated a normally distributed error both on the level of schools and on the level of students while the fixed effect model has the error just on the level of students. Especially this means that each school has it's own level that is not connected to the other levels by a common distribution. This also means that the fixed model does not allow extrapolating to a student of school not included in the original data while the random effect model does so, with an variability that is the sum of the student level and the school level variability. If you are specifically interested in the likelihood we could work that in.

  • 1
    $\begingroup$ (+1) A great answer, which is surprisingly under-voted. I noticed a confusing typo: "excluded" should read "included". Apart from that: what would be an expected practical difference between treating school as random vs. fixed effect? I understand that treating as fixed would not allow predicting a performance of student from a new school, but what about the differences on available data? Let's say other fixed effects are students' gender, race, and weight (whatever). Does treating school as random/fixed influence the power of main effects or interactions of interest? Any other differences? $\endgroup$ – amoeba Feb 26 '14 at 23:40
  • 3
    $\begingroup$ @amoeba Leaving consistency aside, MSE on a student level coefficient can be either more or less efficient in a random vs a fixed effects model depending on, among other things, the level of correlation between student X and the random effect, cluster numbers, etc. Clark and Linzer 2012 has simulation results. $\endgroup$ – conjugateprior May 4 '16 at 17:00
  • 1
    $\begingroup$ @conjugateprior Wow, thanks a lot for this comment! I read the linked paper and it is the most clear explanation of the issue that I have seen. I have spent some considerable amount of time reading various threads here on CV about fixed/random effects, but could not figure out when one should use one over another and why. Reading C&L made a lot of things much clearer to me. Do you perhaps want to write an answer somewhere on CV presenting the summary of this and/or related papers? I am running a bounty on the most voted [mixed-model] thread and will be happy to award you another one there too. $\endgroup$ – amoeba May 4 '16 at 21:21
  • $\begingroup$ @Erik, I edited to correct "partial schooling" to "partial pooling". I think it was a typo but apologies if it was an intended pun! $\endgroup$ – amoeba May 4 '16 at 21:22

In econ land, such effects are individual-specific intercepts (or constants) that are unobserved, but can be estimated using panel data (repeated observation on the same units over time). The fixed effects estimation method allows for correlation between the unit-specific intercepts and the independent explanatory variables. The random effects does not. The cost of using the more flexible fixed effects is that you cannot estimate the coefficient on variables that are time-invariant (like gender, religion, or race).

N.B. Other fields have their own terminology, which can be rather confusing.

  • $\begingroup$ (-1) this says nothing about the mathematical difference between fixed and random effects $\endgroup$ – Macro Apr 10 '12 at 22:08
  • 1
    $\begingroup$ @Macro Agreed. Before that comes up, it would be helpful to know if the econ terminology is what the OP is looking for. I should have been clearer on that. $\endgroup$ – Dimitriy V. Masterov Apr 10 '12 at 22:15
  • $\begingroup$ OK. In that case this may be more appropriate as a comment, wouldn't you say? $\endgroup$ – Macro Apr 10 '12 at 22:15
  • $\begingroup$ The statement "The cost of using the more flexible fixed effects is that you cannot estimate the coefficient on variables that are time-invariant" just isn't true. I just did a simulation where you have repeated measurements on individuals and a single binary predictor that is not time varying. If you include a fixed effect for ID and one for the binary predictor, you most certainly can estimate the coefficient on the binary predictor (although, I will admit, if you don't have a lot of repeated measurements, the estimate does have a large standard error). $\endgroup$ – Macro Apr 10 '12 at 22:45
  • 3
    $\begingroup$ Andrew Gelman (who is not an economist), lists 5 distinct definitions in his ANOVA paper: stat.columbia.edu/~gelman/research/published/banova7.pdf. $\endgroup$ – Dimitriy V. Masterov Apr 10 '12 at 23:07

In a standard software package (e.g. R's lmer), the basic difference is:

  • fixed effects are estimated by maximum likelihood (least squares for a linear model)
  • random effects are estimated by empirical Bayes (least squares with some shrinkage for a linear model, where the shrinkage parameter is chosen by maximum likelihood)

If you're being Bayesian (e.g. WinBUGS), then there is no real difference.

  • 4
    $\begingroup$ I strongly disagree about there being no difference. You could fit a bayesian fixed effects model with all the coefficients having separate priors or a bayesian mixed model where there are hyperparameters. $\endgroup$ – Erik Nov 28 '13 at 15:33
  • $\begingroup$ If you're being Bayesian the difference looks like this. $\endgroup$ – conjugateprior May 14 '16 at 15:11
  • $\begingroup$ @Simon it is an accurate and crispy answer. I should have mentionec it long back. $\endgroup$ – Subhash C. Davar Jun 10 '17 at 16:04

From reading the answers above I guess the major difference is whether we assume a Gaussian for the individual means. Fixed effects don't say much about that assumption because what we are interested is whether A sample differs from B sample (e.g., Are males taller than females?). While if that's not our aim, estimation of individual means can be sometimes meaning less. E.g., 10 people tested in two conditions, and the absolute value of the 20 means are meaning less, because the participants were sampled - what we really interested in is whether the two conditions differ. And then we assume that the individual means are drawn from a Gaussian. And that answers why we should turn to fixed effects when every level is drawn from from the factor - because it is no longer reasonable to assume a hypothetical distribution when the actual distribution is given. I admit that I don't know much about the math behind the calculations.


@Joke A fixed-effects model implies that the effect-size generated by a study(or experiment) is fixed i.e. repeat measurements for an intervention turn out same effect-size.Presumably, the external and internal conditions for the experiment do not change. If you have a number of trials and or studies under different condtions, you will have different effect-sizes. The parametric estimates of mean and variance for a set of effect-sizes can be realised by either presuming that these are fixed-effects or these are random-effects(realised from a super-population). I think that it is matter that can be resolved with the help of mathematical statistics.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.