I am trying to expand my knowledge of statistics. I come from a physical sciences background with a "recipe based" approach to statistical testing, where we say is it continuous, is it normally distributed -- OLS regression.

In my reading I have come across the terms: random effects model, fixed effects model, marginal model. My questions are:

  • In very simple terms, what are they?
  • What are the differences between them?
  • Are any of them synonyms?
  • Where do the traditional tests like OLS regression, ANOVA and ANCOVA fall in this classification?

Just trying to decide where to go next with the self study.

  • 6
  • @gung: The answer you are going to award the bounty to actually by far surpasses all the answers in the "main" thread on the differences between fixed/random effects (linked in the comment above). That question has over 40 upvotes and an accepted answer with 25 upvotes, which unfortunately is nevertheless not very helpful. Should we maybe merge these threads? I guess this would mean that the OP N26 will lose the question upvotes, but their account does not seem to be active anymore anyway. Not sure what's the best course of action. – amoeba Nov 19 '14 at 20:15
  • Thanks @amoeba, I think this deserves more attention, too. It seems to me that that question, while titled similarly, is actually slightly different (& perhaps mis-titled). I don't have the authority to merge these. I did just add a comment there linking to this thread. Why not raise the question of what to do w/ these threads on meta.CV & we'll see what people think? – gung Nov 19 '14 at 20:22
up vote 50 down vote accepted
+50

This question has been partially discussed at this site as below, and opinions seem mixed.

All terms are generally related to longitudinal / panel / clustered / hierarchical data and repeated measures (in the format of advanced regression and ANOVA), but have multiple meanings in different context. I would like to answer the question in formulas based on my knowledge.

Fixed-effects model

  • In biostatistics, fixed-effects, denoted as $\color{red}{\boldsymbol\beta}$ in Equation (*) below, usually comes together with random effects. But the fixed-effects model is also defined to assume that the observations are independent, like cross-sectional setting, as in Longitudinal Data Analysis of Hedeker and Gibbons (2006).
  • In econometrics, the fixed-effects model can be written as $$ y_{ij}=\boldsymbol x_{ij}^{'}\boldsymbol\beta+\color{red}{u_i}+\epsilon_{ij}$$ where $\color{red}{u_i}$ is fixed (not random) intercept for each subject ($i$), or we can also have a fixed-effect as $u_j$ for each repeated measurement ($j$); $\boldsymbol x_{ij}$ denotes covariates.
  • In meta-analysis, the fixed-effect model assumes underlying effect is the same across all studies (e.g. Mantel and Haenszel, 1959).

Random-effects model

  • In biostatistics, the random-effects model (Laird and Ware, 1982) can be written as $$\tag{*} y_{ij}=\boldsymbol x_{ij}^{'}\color{red}{\boldsymbol\beta}+\boldsymbol z_{ij}^{'}\color{blue}{\boldsymbol u_i}+e_{ij}$$ where $\color{blue}{\boldsymbol u_i}$ is assumed to follow a distribution. $\boldsymbol x_{ij}$ denotes covariates for fixed effects, and $\boldsymbol z_{ij}$ denotes covariates for random effects.
  • In econometrics, the random-effects model may only refer to random intercept model as in biostatistics, i.e. $\boldsymbol z_{ij}^{'}=1$ and $\boldsymbol u_i$ is a scalar.
  • In meta-analysis, the random-effect model assumes heterogeneous effects across studies (DerSimonian and Laird, 1986).

Marginal model

Marginal model is generally compared to conditional model (random-effects model), and the former focuses on the population mean (take linear model for an example) $$E(y_{ij})=\boldsymbol x_{ij}^{'}\boldsymbol\beta,$$ while the latter deals with the conditional mean $$E(y_{ij}|\boldsymbol u_i)=\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i.$$ The interpretation and scale of the regression coefficients between marginal model and random-effects model would be different for nonlinear models (e.g. logistic regression). Let $h(E(y_{ij}|\boldsymbol u_i))=\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i$, then $$E(y_{ij})=E(E(y_{ij}|\boldsymbol u_i))=E(h^{-1}(\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i))\neq h^{-1}(\boldsymbol x_{ij}^{'}\boldsymbol\beta),$$ unless trivially the link function $h$ is the identity link (linear model), or $u_i=0$ (no random-effects). Good examples include generalized estimating equations (GEE; Zeger, Liang and Albert, 1988) and marginalized multilevel models (Heagerty and Zeger, 2000).

  • +1 (long time ago), but I am a bit confused by your first bullet in the Fixed Effects part. fixed-effects usually comes together with random effects, denoted as β in the second formula -- what is denoted as beta? which second formula? can you give an example formula in the same bullet? It would be nice to have a formula for fixed effects in statistics, fixed effects in econometrics, random effects in statistics, and random effects in econometrics. Thanks a lot. – amoeba Oct 23 '15 at 14:16
  • Thanks for your suggestions, @amoeba! I tagged the "second formula" as equation (*). The fixed effects are colored as red, and random effects as blue. Hope it's cleared right now. – Jiebiao Wang Oct 23 '15 at 14:55
  • 1
    @skan $\boldsymbol z_{ij}$ denotes covariates for random effects. It is a vector, and $\boldsymbol z_{ij}^{'}$ is the transpose. – Jiebiao Wang Mar 3 '16 at 21:41
  • 1
    Here is a detailed example. Hope it helps. @skan – Jiebiao Wang Apr 24 '17 at 22:13
  • 1
    @skan It is not suggested to have both, either is enough. Here is a perfect example. – Jiebiao Wang Apr 25 '17 at 13:34

Correct me if I'm wrong here:

Conceptually, there are four possible effects: Fixed intercept, fixed coefficient, random intercept, random coefficient. Most regression models are 'random effects', so they have random intercepts and random coefficients. The term 'random effect' came into use in contrast to 'fixed effect'.

'Fixed effect' is when a variable effects some of the sample, but not all. The simplest version of a fixed effect model (conceptually) would be a dummy variable, for a fixed effect with a binary value. These models have a single random intercept, fixed effect coefficients, and random variable coefficients.

The next tier of complication (conceptually) is when the fixed effect is not binary, but nominal, with many values. In this case, what is generated is a model with many intercepts (one for each of the nominal values). This is where you get the classic 'multiple lines' of a panel data model, where each of the 'options' of a fixed effect variable gets its own effect. The virtue of throwing all the different factor-specific data series into a single regression (rather than doing each factor of the fixed effect as its own regression) is that you get to pool the variance of all the different effects in one equation, and so get better (more certain) values for all of your coefficients.

'Tier three' of complication would be when the 'fixed effect' is itself a random variable, except that its effects are 'fixed' to affect only a sub-set of the sample. At which point the model would have a random intercept, multiple fixed intercepts, and multiple random variables. I think this is what is known as a 'mixed effects' model?

'Mixed effect' models get used for multi-level modeling (MLM), as the 'fixed effects' can be used for nesting one subset of data within another. This grouping can have multiple tiers, with students nested within classrooms, nested within schools. The school is a fixed effect on the classrooms, and the classrooms on the students. (The school may or may not be a fixed effect on the student, depending on the experimental design--not sure)

Panel data models are 'mixed effect' models, but use two dimensions for grouping, typically time and some sort of category.

  • Not sure what you mean by "Fixed effects cover 'sets' of choices: A or B; ... Random effects include things like body weight". Do you mean fixed effects are for discrete variables, random effects are for continuous variables? Also not sure about why "using multiple dummy variables for the same thing is statistically inappropriate". The fixed effects model in econometrics does have a dummy variable for each "panel". I cannot agree with "'Mixed' models ... Having 'fixed' intercepts by grouping, they no longer have a random intercept" either. Many mixed-effects models have a random intercept. – Jiebiao Wang Oct 24 '16 at 20:12
  • My understanding is imperfect. I will edit my response and try again. – Mox Oct 25 '16 at 17:13
  • Is it possible that a variable appear simultaenously as fixed effect and as random effect? – skan Apr 24 '17 at 10:45

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.