61
$\begingroup$

I am trying to expand my knowledge of statistics. I come from a physical sciences background with a "recipe based" approach to statistical testing, where we say is it continuous, is it normally distributed -- OLS regression.

In my reading I have come across the terms: random effects model, fixed effects model, marginal model. My questions are:

  • In very simple terms, what are they?
  • What are the differences between them?
  • Are any of them synonyms?
  • Where do the traditional tests like OLS regression, ANOVA and ANCOVA fall in this classification?

Just trying to decide where to go next with the self study.

$\endgroup$
3
  • 9
    $\begingroup$ Of possible interest: What is the difference between fixed effect, random effect and mixed effect models? $\endgroup$
    – chl
    Jan 26, 2012 at 13:16
  • 2
    $\begingroup$ @gung: The answer you are going to award the bounty to actually by far surpasses all the answers in the "main" thread on the differences between fixed/random effects (linked in the comment above). That question has over 40 upvotes and an accepted answer with 25 upvotes, which unfortunately is nevertheless not very helpful. Should we maybe merge these threads? I guess this would mean that the OP N26 will lose the question upvotes, but their account does not seem to be active anymore anyway. Not sure what's the best course of action. $\endgroup$
    – amoeba
    Nov 19, 2014 at 20:15
  • 2
    $\begingroup$ Thanks @amoeba, I think this deserves more attention, too. It seems to me that that question, while titled similarly, is actually slightly different (& perhaps mis-titled). I don't have the authority to merge these. I did just add a comment there linking to this thread. Why not raise the question of what to do w/ these threads on meta.CV & we'll see what people think? $\endgroup$ Nov 19, 2014 at 20:22

2 Answers 2

70
+50
$\begingroup$

This question has been partially discussed at this site as below, and opinions seem mixed.

All terms are generally related to longitudinal / panel / clustered / hierarchical data and repeated measures (in the format of advanced regression and ANOVA), but have multiple meanings in different context. I would like to answer the question in formulas based on my knowledge.

Fixed-effects model

  • In biostatistics, fixed-effects, denoted as $\color{red}{\boldsymbol\beta}$ in Equation (*) below, usually comes together with random effects. But the fixed-effects model is also defined to assume that the observations are independent, like cross-sectional setting, as in Longitudinal Data Analysis of Hedeker and Gibbons (2006).
  • In econometrics, the fixed-effects model can be written as $$ y_{ij}=\boldsymbol x_{ij}^{'}\boldsymbol\beta+\color{red}{u_i}+\epsilon_{ij}$$ where $\color{red}{u_i}$ is fixed (not random) intercept for each subject ($i$), or we can also have a fixed-effect as $u_j$ for each repeated measurement ($j$); $\boldsymbol x_{ij}$ denotes covariates.
  • In meta-analysis, the fixed-effect model assumes underlying effect is the same across all studies (e.g. Mantel and Haenszel, 1959).

Random-effects model

  • In biostatistics, the random-effects model (Laird and Ware, 1982) can be written as $$\tag{*} y_{ij}=\boldsymbol x_{ij}^{'}\color{red}{\boldsymbol\beta}+\boldsymbol z_{ij}^{'}\color{blue}{\boldsymbol u_i}+e_{ij}$$ where $\color{blue}{\boldsymbol u_i}$ is assumed to follow a distribution. $\boldsymbol x_{ij}$ denotes covariates for fixed effects, and $\boldsymbol z_{ij}$ denotes covariates for random effects.
  • In econometrics, the random-effects model may only refer to random intercept model as in biostatistics, i.e. $\boldsymbol z_{ij}^{'}=1$ and $\boldsymbol u_i$ is a scalar.
  • In meta-analysis, the random-effect model assumes heterogeneous effects across studies (DerSimonian and Laird, 1986).

Marginal model

Marginal model is generally compared to conditional model (random-effects model), and the former focuses on the population mean (take linear model for an example) $$E(y_{ij})=\boldsymbol x_{ij}^{'}\boldsymbol\beta,$$ while the latter deals with the conditional mean $$E(y_{ij}|\boldsymbol u_i)=\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i.$$ The interpretation and scale of the regression coefficients between marginal model and random-effects model would be different for nonlinear models (e.g. logistic regression). Let $h(E(y_{ij}|\boldsymbol u_i))=\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i$, then $$E(y_{ij})=E(E(y_{ij}|\boldsymbol u_i))=E(h^{-1}(\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i))\neq h^{-1}(\boldsymbol x_{ij}^{'}\boldsymbol\beta),$$ unless trivially the link function $h$ is the identity link (linear model), or $u_i=0$ (no random-effects). Good examples include generalized estimating equations (GEE; Zeger, Liang and Albert, 1988) and marginalized multilevel models (Heagerty and Zeger, 2000).

$\endgroup$
14
  • 1
    $\begingroup$ Thanks, Randel. One more question, about "mixed model" terminology. As far as I understand, in biostatistics your equation (*) would be called a mixed model because it contains both random and fixed effects. Is that correct? But is the term "mixed model" also used in econometrics? If so, what does it refer to? $\endgroup$
    – amoeba
    Oct 23, 2015 at 15:02
  • $\begingroup$ Yes, equation (*) is also called a mixed model in (bio)statistics. As far as I know, econometrist may not call it "mixed model", but "random-effects model" or "random coefficient model", if they are interested in cluster heterogeneity. To me, the only difference is the assumption for cluster-specific effect, fixed or random. $\endgroup$
    – Randel
    Oct 23, 2015 at 15:11
  • 1
    $\begingroup$ @skan $\boldsymbol z_{ij}$ denotes covariates for random effects. It is a vector, and $\boldsymbol z_{ij}^{'}$ is the transpose. $\endgroup$
    – Randel
    Mar 3, 2016 at 21:41
  • 1
    $\begingroup$ Here is a detailed example. Hope it helps. @skan $\endgroup$
    – Randel
    Apr 24, 2017 at 22:13
  • 1
    $\begingroup$ @skan It is not suggested to have both, either is enough. Here is a perfect example. $\endgroup$
    – Randel
    Apr 25, 2017 at 13:34
1
$\begingroup$

Correct me if I'm wrong here:

Conceptually, there are four possible effects: Fixed intercept, fixed coefficient, random intercept, random coefficient. Most regression models are 'random effects', so they have random intercepts and random coefficients. The term 'random effect' came into use in contrast to 'fixed effect'.

'Fixed effect' is when a variable effects some of the sample, but not all. The simplest version of a fixed effect model (conceptually) would be a dummy variable, for a fixed effect with a binary value. These models have a single random intercept, fixed effect coefficients, and random variable coefficients.

The next tier of complication (conceptually) is when the fixed effect is not binary, but nominal, with many values. In this case, what is generated is a model with many intercepts (one for each of the nominal values). This is where you get the classic 'multiple lines' of a panel data model, where each of the 'options' of a fixed effect variable gets its own effect. The virtue of throwing all the different factor-specific data series into a single regression (rather than doing each factor of the fixed effect as its own regression) is that you get to pool the variance of all the different effects in one equation, and so get better (more certain) values for all of your coefficients.

'Tier three' of complication would be when the 'fixed effect' is itself a random variable, except that its effects are 'fixed' to affect only a sub-set of the sample. At which point the model would have a random intercept, multiple fixed intercepts, and multiple random variables. I think this is what is known as a 'mixed effects' model?

'Mixed effect' models get used for multi-level modeling (MLM), as the 'fixed effects' can be used for nesting one subset of data within another. This grouping can have multiple tiers, with students nested within classrooms, nested within schools. The school is a fixed effect on the classrooms, and the classrooms on the students. (The school may or may not be a fixed effect on the student, depending on the experimental design--not sure)

Panel data models are 'mixed effect' models, but use two dimensions for grouping, typically time and some sort of category.

$\endgroup$
5
  • $\begingroup$ Not sure what you mean by "Fixed effects cover 'sets' of choices: A or B; ... Random effects include things like body weight". Do you mean fixed effects are for discrete variables, random effects are for continuous variables? Also not sure about why "using multiple dummy variables for the same thing is statistically inappropriate". The fixed effects model in econometrics does have a dummy variable for each "panel". I cannot agree with "'Mixed' models ... Having 'fixed' intercepts by grouping, they no longer have a random intercept" either. Many mixed-effects models have a random intercept. $\endgroup$
    – Randel
    Oct 24, 2016 at 20:12
  • 1
    $\begingroup$ My understanding is imperfect. I will edit my response and try again. $\endgroup$
    – Mox
    Oct 25, 2016 at 17:13
  • $\begingroup$ Is it possible that a variable appear simultaenously as fixed effect and as random effect? $\endgroup$
    – skan
    Apr 24, 2017 at 10:45
  • $\begingroup$ <theanalysisfactor.com/mixed-models-predictor-both-fixed-random> says yes. $\endgroup$
    – Mox
    May 1, 2017 at 22:19
  • $\begingroup$ @skan: it can give fixed effect in "between-groups" analysis & random effect in "within-group" analysis == meaning mixed-effects models. Use GLMM to regress such vars OR ANCOVA for dispersion analysis $\endgroup$
    – JeeyCi
    Jan 2 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.