13
$\begingroup$

Bolker (2015) writes on p. 313 that

Frequentists and Bayesians define random effects somewhat differently, which affects the way they use them. Frequentists define random effects as categorical variables whose levels are chosen at random from a larger population, e.g., species chosen at random from a list of endemic species. Bayesians define random effects as sets of variables whose parameters are [all] drawn from [the same] distribution. The frequentist definition is philosophically coherent, and you will encounter researchers (including reviewers and supervisors) who insist on it, but it can be practically problematic. For example, it implies that you can’t use species as random effect when you have observed all of the species at your field site—since the list of species is not a sample from a larger population—or use year as a random effect, since researchers rarely run an experiment in randomly sampled years—they usually use either a series of consecutive years, or the haphazard set of years when they could get into the field.

Bolker continues on p 315 to state that

The Bayesian framework has a simpler definition of random effects. Under a Bayesian approach, a fixed effect is one where we estimate each parameter (e.g., the mean for each species within a genus) independently (with independently specified priors), while for a random effect the parameters for each level are modeled as being drawn from a distribution (usually Normal); in standard statistical notation, $\textrm{species_mean} \sim {\cal N}(\textrm{genus_mean}, \sigma^2_{\textrm{species}})$.

Bolker's chapter thus provides clear Bayesian and Frequentist definitions of random effects, and a clear Bayesian definition of fixed effects. However, I don't see that it provides any Frequentist definition of fixed effects.

I am aware from this answer that a wide variety of inconsistent definitions of fixed effects exist in the literature. To clarify, I'm looking for a definition that would "complete the set" of Bolker's existing ones, and be demonstrably consistent with the approach that he is taking.

Although related topics are considered at length in this question, I see this question as different and much more specific. I also don't think the answer is present in any of the responses to the other question.

Bolker, B. M., 2015. Linear and generalized linear mixed models. In G. A. Fox, S. Negrete-Yankelevich, and V. J. Sosa (eds.), Ecological Statistics: Contemporary theory and application. Oxford University Press. ISBN 978-0-19-967255-4. In press.

$\endgroup$
  • 3
    $\begingroup$ I am puzzled by this question because in its background it focuses exclusively on the definition of random effects and then, only at the very end, asks about fixed effects! A suitable point of departure would be a textbook definition--most likely that would resolve your question easily. $\endgroup$ – whuber May 11 at 12:01
  • 1
    $\begingroup$ This is a fair point, but my trouble stems not from a lack of definitions but rather from a profusion of contradictory ones. For instance, this answer lists five varying definitions. I'd tended to conceptualize fixed effects not in any of those five ways, but rather as being exhaustive in the sense that the levels of the categorical variable in question are the only ones I'm interested in. However, I'm not sure if that's consistent with @ben-bolker's approach. $\endgroup$ – user1205901 - Reinstate Monica May 11 at 12:22
  • $\begingroup$ I've edited the OP to clarify that I'm hoping for a definition consistent with @ben-bolker's Bayesian definitions of fixed/random effects and his Frequentist definition of random effects. $\endgroup$ – user1205901 - Reinstate Monica May 11 at 12:22
  • 2
    $\begingroup$ I find three characterizations of fixed effects in that answer and they don't contradict each other. Moreover, why should characterizations of any concept have to be exhaustive? (Indeed, any generally useful concept will have many different characterizations.) If you want a clear definition, then look at the mathematics: a fixed effect is unambiguously defined by the model in which it appears. $\endgroup$ – whuber May 11 at 12:30
  • 1
    $\begingroup$ "Frequentists and Bayesians define...." It is wrong to speak about Frequentist and Bayesian as groups of different people (with different believes about probability). It is right that a certain field one approach might be more common than the other approach. But that is all pragmatic, and not so much philosophical (and not based on personal believes about probability). A carpenter uses mostly a hammer and an electrician mostly a screwdriver, but that doesn't make these two groups opposed to each other and the one tool better than the other, it just depends on circumstances. $\endgroup$ – Sextus Empiricus May 11 at 13:41
3
$\begingroup$

First of all, the 'random effects' can be viewed in different ways and the approaches to them and associated definitions may seem conflicting but it is just a different viewpoint.

The 'random effect' term in a model can be seen as both a term in the deterministic part of the model as a term in the random part of the model.

Basically, in general, the difference between fixed effect and random effect is whether a parameter is considered fixed within the experiment or not. From that point you get all kinds of different practical applications, and the many varying answers (opinions) to the question "When to use random effects?". It might actually be more a linguistic problem (when something is called random effect or not) than something with a problem with modelling (where we all understand the mathematics in the same way).


The Bayesian and Frequentist frameworks look in the same way at a statistical model, say: observations $Y_{ij}$ where $j$ is the observation number and $i$ indicates a grouping

$$Y_{ij} = \underbrace{ \alpha + \beta}_{\substack{\llap{\text{mod}}\rlap{\text{el}} \\ \llap{\text{parame}}\rlap{\text{ters}} }}\overbrace{X_{ij}}^{\substack{\llap{\text{indep}}\rlap{\text{endent}} \\ \text{variables}}} +\overbrace{Z_{i}}^{\substack{\llap{\text{ran}}\rlap{\text{dom}} \\ \text{group}\\ \text{term}}} + \overbrace{\epsilon_{j}}^{\substack{\llap{\text{ran}}\rlap{\text{dom}} \\ \text{individual}\\ \text{term}}}$$

The observations $Y_{ij}$ will depend on some model parameters $\alpha$ and $\beta$, which can be seen as the 'effects' which describe how the $Y_{ij}$ depends on the variable $X_{ij}$.

But the observations will not be deterministic and only depend on $X_{ij}$, there will also be random terms such that the observation conditional on the independent variables $Y_{ij} \vert X_{ij}$ will follow some random distribution. The terms $Z_{i}$ and $\epsilon_j$ are the nondeterministic part of the model.

This is the same for the Bayesian and Frequentist approach, which in principle do not differ in their way to describe a probability for the observations $Y_{ij}$ conditional on the model parameters $\alpha$ and $\beta$ and independent variables $X_{ij}$, where $Z_i$ and $\epsilon_j$ describe a non-deterministic part.


The difference is in the approach to 'inference'.

  • The Bayesian approach uses reverse probability and describes a probability distribution of the (fixed effect) parameters $\alpha$ and $\beta$. This implies an interpretation of those parameters as random variables. With a Bayesian approach the outcome is a statement about the probability distribution for the fixed effect parameters $\alpha$ and $\beta$.

  • A Frequentist method does not consider a distribution of the fixed effect parameters $\alpha$ and $\beta$ and avoids making statements that imply such distribution (but it is not explicitly rejected). The probability/frequency statements in a frequentist approach do not relate to a frequency/probability statement about the parameters but to a frequency/probability statement about the success rate of the estimation procedure.

So if you like, you could say that a frequentist definition of a fixed effect is: 'a model parameter that describes the deterministic part in a statistical model'. (ie. parameters that describe how dependent variables depend on independent variables).

And more specifically in most contexts this relates only to the parameters for the deterministic model that describe $E[Y_{ij} \vert X_{ij}]$. For instance, with a frequentist model one can estimate both the mean and variance, but only the parameters that relate to the mean are considered 'effects'. And even more specifically, the effects are most often used in the context of a 'linear' model. E.g. a for a nonlinear model like $E[y] \sim a e^{-bt}$ the parameters $a$ and $b$ are not really called 'effects'.


In a Bayesian framework all effects are sort of random and not deterministic (so the difference between random effect and fixed effect is not so obvious). The model parameters $\alpha$ and $\beta$ are random variables.

How I interpret the question's description/definition of the difference in random effect and fixed effect in the Bayesian framework is more as something pragmatic than as some principle.

  • the fixed effects $\alpha$ and $\beta$ are considered to be like "where we estimate each parameter ... independently" (the $\alpha$ and $\beta$ are randomly drawn from a distribution, but they are the same for all $i$ and $j$ within the analysis, e.g. the mean of a species is a model parameter that is considered the same for each species)
  • and the random effects are like "for a random effect the parameters for each level are modeled as being drawn from a distribution" (for each observation category $i$ a different random effect is 'drawn' from the distribution, e.g. the mean of a species is a model parameter that is considered different for each species)

In a frequentist framework the fixed effect model parameters are not considered as random parameters, or at least it doesn't matter for the inference whether the parameters are a random parameter or not and it is left out the analysis. However, the random effect term is explicitly considered as a random variable (that is, as a nondeterministic component of the model) and this will influence the analysis (e.g. as in a mixed effects model the imposed structure of the random error term).

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Basically, the difference between fixed effect and random effect is whether a parameter is considered fixed within the experiment or not. From that point you get all kinds of different practical applications, the many answers to the question "When to use random effects?". $\endgroup$ – Sextus Empiricus May 12 at 12:02
5
$\begingroup$

Trying to find single "authoritative" definition is always tempting in cases like this, but the variety of different definitions shows that this term simply is not used in consistent manner. Andrew Gelman seems to have reached same conclusions, you can look as his blog posts here and here, or into his handbook Data Analysis Using Regression and Multilevel/Hierarchical Models written together with Jennifer Hill, where they write (p. 254-255):

The term fixed effects is used in contrast to random effects—but not in a consistent way! Fixed effects are usually defined as varying coefficients that are not themselves modeled. For example, a classical regression including $J − 1 = 19$ city indicators as regression predictors is sometimes called a “fixed-effects model” or a model with “fixed effects for cities.” Confusingly, however, “fixed-effects models” sometimes refer to regressions in which coefficients do not vary by group (so that they are fixed, not random).

A question that commonly arises is when to use fixed effects (in the sense of varying coefficients that are unmodeled) and when to use random effects. The statistical literature is full of confusing and contradictory advice. Some say that fixed effects are appropriate if group-level coefficients are of interest, and random effects are appropriate if interest lies in the underlying population. Others recommend fixed effects when the groups in the data represent all possible groups, and random effects when the population includes groups not in the data. These two recommendations (and others) can be unhelpful. For example, in the child support example, we are interested in these particular cities and also the country as a whole. The cities are only a sample of cities in the United States—but if we were suddenly given data from all the other cities, we would not want then to change our model.

Our advice (elaborated upon in the rest of this book) is to always use multilevel modeling (“random effects”). Because of the conflicting definitions and advice, we avoid the terms “fixed” and “random” entirely, and focus on the description of the model itself (for example, varying intercepts and constant slopes), with the understanding that batches of coefficients (for example, $\alpha_1, \alpha_2, \dots, \alpha_J$) will themselves be modeled.

This is a good advice.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Although I take the point that searching for a single authoritative definition is futile here, do you think it is also futile to search for a definition that would at least "match" (be consistent with) Bolker's three definitions? $\endgroup$ – user1205901 - Reinstate Monica May 11 at 23:05
  • $\begingroup$ @user1205901-ReinstateMonica the answer is that I’m not sure if such definition exists, those a are ambiguously used terms. $\endgroup$ – Tim May 12 at 5:39
  • $\begingroup$ To add yet another option (to the fixed effects versus random effects menu), consider 4.3 The Group Lasso in Statistical Learning with Sparsity: The Lasso and Generalizations (web.stanford.edu/~hastie/StatLearnSparsity_files/SLS.pdf), which I think of (in a very hand-wavy way) as being somewhere in between fixed effects and random effects. $\endgroup$ – Adrian May 13 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.