What is a difference between random effects-, fixed effects- and marginal model?

Question

I am trying to expand my knowledge of statistics. I come from a physical sciences background with a "recipe based" approach to statistical testing, where we say is it continuous, is it normally distributed -- OLS regression.

In my reading I have come across the terms: random effects model, fixed effects model, marginal model. My questions are:

• In very simple terms, what are they?
• What are the differences between them?
• Are any of them synonyms?
• Where do the traditional tests like OLS regression, ANOVA and ANCOVA fall in this classification?

Just trying to decide where to go next with the self study.

Answer 1

This question has been partially discussed at this site as below, and opinions seem mixed.

• What is the difference between fixed effect, random effect and mixed effect models?
• What is the mathematical difference between random- and fixed-effects?
• Concepts behind fixed/random effects models

All terms are generally related to longitudinal / panel / clustered / hierarchical data and repeated measures (in the format of advanced regression and ANOVA), but have multiple meanings in different context. I would like to answer the question in formulas based on my knowledge.

Fixed-effects model

• In biostatistics, fixed-effects, denoted as $\color{red}{\boldsymbol\beta}$ in Equation (*) below, usually comes together with random effects. But the fixed-effects model is also defined to assume that the observations are independent, like cross-sectional setting, as in Longitudinal Data Analysis of Hedeker and Gibbons (2006).
• In econometrics, the fixed-effects model can be written as $$ y_{ij}=\boldsymbol x_{ij}^{'}\boldsymbol\beta+\color{red}{u_i}+\epsilon_{ij}$$ where $\color{red}{u_i}$ is fixed (not random) intercept for each subject ($i$), or we can also have a fixed-effect as $u_j$ for each repeated measurement ($j$); $\boldsymbol x_{ij}$ denotes covariates.
• In meta-analysis, the fixed-effect model assumes underlying effect is the same across all studies (e.g. Mantel and Haenszel, 1959).

Random-effects model

• In biostatistics, the random-effects model (Laird and Ware, 1982) can be written as $$\tag{*} y_{ij}=\boldsymbol x_{ij}^{'}\color{red}{\boldsymbol\beta}+\boldsymbol z_{ij}^{'}\color{blue}{\boldsymbol u_i}+e_{ij}$$ where $\color{blue}{\boldsymbol u_i}$ is assumed to follow a distribution. $\boldsymbol x_{ij}$ denotes covariates for fixed effects, and $\boldsymbol z_{ij}$ denotes covariates for random effects.
• In econometrics, the random-effects model may only refer to random intercept model as in biostatistics, i.e. $\boldsymbol z_{ij}^{'}=1$ and $\boldsymbol u_i$ is a scalar.
• In meta-analysis, the random-effect model assumes heterogeneous effects across studies (DerSimonian and Laird, 1986).

Marginal model

Marginal model is generally compared to conditional model (random-effects model), and the former focuses on the population mean (take linear model for an example) $$E(y_{ij})=\boldsymbol x_{ij}^{'}\boldsymbol\beta,$$ while the latter deals with the conditional mean $$E(y_{ij}|\boldsymbol u_i)=\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i.$$ The interpretation and scale of the regression coefficients between marginal model and random-effects model would be different for nonlinear models (e.g. logistic regression). Let $h(E(y_{ij}|\boldsymbol u_i))=\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i$, then $$E(y_{ij})=E(E(y_{ij}|\boldsymbol u_i))=E(h^{-1}(\boldsymbol x_{ij}^{'}\boldsymbol\beta + \boldsymbol z_{ij}^{'}\boldsymbol u_i))\neq h^{-1}(\boldsymbol x_{ij}^{'}\boldsymbol\beta),$$ unless trivially the link function $h$ is the identity link (linear model), or $u_i=0$ (no random-effects). Good examples include generalized estimating equations (GEE; Zeger, Liang and Albert, 1988) and marginalized multilevel models (Heagerty and Zeger, 2000).

Answer 2

Correct me if I'm wrong here:

Conceptually, there are four possible effects: Fixed intercept, fixed coefficient, random intercept, random coefficient. Most regression models are 'random effects', so they have random intercepts and random coefficients. The term 'random effect' came into use in contrast to 'fixed effect'.

'Fixed effect' is when a variable effects some of the sample, but not all. The simplest version of a fixed effect model (conceptually) would be a dummy variable, for a fixed effect with a binary value. These models have a single random intercept, fixed effect coefficients, and random variable coefficients.

The next tier of complication (conceptually) is when the fixed effect is not binary, but nominal, with many values. In this case, what is generated is a model with many intercepts (one for each of the nominal values). This is where you get the classic 'multiple lines' of a panel data model, where each of the 'options' of a fixed effect variable gets its own effect. The virtue of throwing all the different factor-specific data series into a single regression (rather than doing each factor of the fixed effect as its own regression) is that you get to pool the variance of all the different effects in one equation, and so get better (more certain) values for all of your coefficients.

'Tier three' of complication would be when the 'fixed effect' is itself a random variable, except that its effects are 'fixed' to affect only a sub-set of the sample. At which point the model would have a random intercept, multiple fixed intercepts, and multiple random variables. I think this is what is known as a 'mixed effects' model?

'Mixed effect' models get used for multi-level modeling (MLM), as the 'fixed effects' can be used for nesting one subset of data within another. This grouping can have multiple tiers, with students nested within classrooms, nested within schools. The school is a fixed effect on the classrooms, and the classrooms on the students. (The school may or may not be a fixed effect on the student, depending on the experimental design--not sure)

Panel data models are 'mixed effect' models, but use two dimensions for grouping, typically time and some sort of category.