Relative variances of higher-order vs. lower-order random terms in mixed models

Question

TL, DR summary:

Is there any theoretical or empirical basis to support the following statement being true as a general rule of thumb?

"When estimating a mixed model, typically the estimated variances/standard deviations of random effects associated with 'higher-order' terms (e.g., random effects of two-way, three-way, and beyond interaction terms) turn out to be smaller than the estimated variances/standard deviations of random effects associated with 'lower-order' terms (e.g., the residual variance, variances associated with simple effects of grouping factors)."

The source of this claim is me. ;)

---

Okay, now for the longer version ...

Typically when I sit down to start analyzing a new dataset which I know will call for a mixed model, one of the first models that I fit (after the statistical foreplay of looking through the observations in the dataset, plotting various things, cross-tabulating different factors, etc.) is one that is pretty close to the "maximal" random effects specification, where every random effect that is in-principle possible to estimate from the data, is estimated.

Naturally, it is not uncommon that this nearly-maximal model will have some computational problems (convergence errors, or wacky variance/covariance estimates, or etc.) and that I have to trim back this model to find one that my data can more easily support. Fine.

In these situations, the method I have come to prefer for trimming random terms is not to rely on significance tests or likelihood ratios, but rather to just identify the random effects that seem to have the smallest standard deviations (which can admittedly be a little tricky when predictors are on very different scales, but I try to take account of this in my appraisal) and remove these terms first, sequentially in an iterative process. The idea being that I want to alter the predictions of the model as little as possible while still reducing the complexity of the model.

One pattern that I seem to have noticed after a pretty good amount of time spent doing this is that following this method very often leads me to trim random effects associated with higher-order terms (as defined above) of the model first. This is not always true, and occasionally some of the higher-order terms explain a lot of variance, but this doesn't seem to be the general pattern. In sharp contrast, I usually find that lower-order random terms -- particularly those associated with simple effects of the grouping factors -- explain a pretty good amount of variance and are fairly essential to the model. At the extreme, the residual term commonly accounts for close to the most variance, although of course removing this term wouldn't be sensible.

This entirely informal observation leads me to form the hypothesis that I stated at the beginning of this question.

If it is true, then it constitutes a useful piece of advice that might be passed down to people who are less experience with this kind of model selection process. But before I begin doing so, I want to check with other, more experienced users of mixed models about their reactions to this observation. Does it seem more or less true to you? Is it roughly consistent with your experience fitting many different mixed models to many different datasets? Do you know of any sensible, theoretical reasons why we might actually expect this to be true in a lot of cases? Or does it just seem like bullshit?

One possible answer here is that it is not true even in my own case, and I have simply deceived myself. Certainly a possibility that I am open to.

Another possibility is that it might be true in my own case, but that this could simply be a kind of coincidence having to do with the kinds of datasets that I tend to work with routinely (which, FYI, are datasets in psychological / social sciences, a slight majority being experimental in origin, but also a fair proportion of non-experimental stuff). If this is the case then there is probably no good reason for expecting my observations to hold in general in other fields that handle very different kinds of data. Still, if there is a coherent non-coincidental reason for why this might be expected to be true, even if only for these particular kinds of datasets, I would love to hear it.

And of course another possibility is that others have noticed similar patterns in their own data, and that it represents some kind of general rule of thumb that people find useful to keep in mind as they fit mixed models to various different data. If this is the case then it seems like there must be some compelling statistical-theoretical reason for why this pattern arises. But I really don't know what that reason would look like.

I welcome anyone's thoughts and opinions about this. Note that as far as I'm concerned, totally legitimate responses to this question might be as simple as comments like "Yeah I have noticed something similar in the data I've worked with, but I have no idea why it should be true" or conversely "I have noticed nothing remotely like this in the data I've worked with." Of course I also welcome longer and more involved discussions ...

Answer 1

I have discovered that the regularity I described in my question has in fact been written about by several authors in the literature on Design of Experiments (DoE). It has been called the "hierarchical ordering principle" and also sometimes the "sparsity-of-effects principle."

In the chapter on fractional factorial designs in Montgomery (2013, p. 290), he writes:

The successful use of fractional factorial designs is based on three key ideas:

1. The sparsity of effects principle. When there are several variables, the system or process is likely to be driven primarily by some of the main effects and low-order interactions.

...

Wu & Hamada (2000, p. 143) instead call this the "hierarchical ordering principle", and use the phrase "sparsity of effects" to refer to a related but distinct observation:

Three fundamental principles for factorial effects:

Hierarchical ordering principle: (i) Lower order effects are more likely to be important than higher order effects, (ii) effects of the same order are likely to be equally important .

Effect sparsity principle: The number of relatively important effects in a factorial experiment is small.

...

Li, Sudarsanam, & Frey (2006, p. 34) give two possible explanations for why hierarchical ordering should tend to occur. First they suggest that it is "partly due to the range over which experimenters typically explore factors":

In the limit that experimenters explore small changes in factors and to the degree that systems exhibit continuity of responses and their derivatives, linear effects of factors tend to dominate. Therefore, to the extent that hierarchical ordering is common in experimentation, it is due to the fact that many experiments are conducted for the purpose of minor refinement rather than broad-scale exploration

They next suggest that it is "partly determined by the ability of experimenters to transform the inputs and outputs of the system to obtain a parsimonious description of system behavior":

For example, it is well known to aeronautical engineers that the lift and drag of wings is more simply described as a function of wing area and aspect ratio than by wing span and chord. Therefore, when conducting experiments to guide wing design, engineers are likely to use the product of span and chord (wing area) and the ratio of span and chord (the aspect ratio) as the independent variables

References

• Li, X., Sudarsanam, N., & Frey, D. D. (2006). Regularities in data from factorial experiments. Complexity, 11(5), 32-45.
• Montgomery, D. C. (2013). Design and analysis of experiments (Vol. 8). New York: Wiley.
• Wu, C. J., & Hamada, M. S. (2000). Experiments: planning, analysis, and optimization (Vol. 552). John Wiley & Sons.

Answer 2

Blockquote

In general, I agree with the original hypotheses that higher-order terms are often associated with smaller variances. But, this also depends on the type of data.

In plant breeding, a rule of thumb (Gauch, 1996, page 90) for multi-environment trials is that the variation in the data is: 70% location, 20% location-by-variety, 10% variety

Very approximate, but it is fairly consistent that the higher-order term "location-by-variety" variance is larger than the main-effect "variety" variance.

Ref: H G Gauch and R W Zobel, 1996. Book: Genotype by Environment Interaction. Chapter: AMMI analysis of yield trials. CRC Press.