When can we exclude random effects?

Question

Random effects are useful for accounting for variance between groups (e.g., students within schools; patients within hospitals) or avoiding cases of pseudorelication with repeated measures data. But, technically, one could imagine many ways in which data could be grouped even if the intent wasn’t a blocked or repeated measures design.

For e.g.: If you measured every school in the province on some average metric, but didn’t have in your dataset some geographical variable to delineate where these schools were, would one need to worry about this? My point is that there are potentially numerous random effects that could be included in any design (as data can often be seen as grouped), but to me this is different than an explicit multi-leveled sampling design (patients > hospitals > counties > states). At what point does one “draw the line” on including a random effect?

Answer 1

But, technically, one could imagine many ways in which data could be grouped even if the intent wasn’t a blocked or repeated measures design.

True, but the issue isn't about blocking or repeated measures, it's about non-independence of observations within the groups. If there is non-independence, this needs to be accounted for in order to provide valid inferences. Either random intercepts or fixed effects for the grouping variable can do this. If the obervations are independent then it should be safe to exclude random effects, and fixed effects won't be needed either.

But if there is non-independence, then obviously with a lot of groups it is preferable to fit random intercepts, while if there are very few then random intercepts are not appropriate. There is no way to draw the line about how many groups are needed. There is some consensus around 6, but there is also some disagreement on this.

The example in your question:

If you measured every school in the province on some average metric, but didn’t have in your dataset some geographical variable to delineate where these schools were, would one need to worry about this?

So just to generalise this, you are talking about the absence of cluster-level variables. A mixed effects / multilevel model is still warranted in such circumstances. Even a variance components model (with no fixed effects whatsoever) can be a useful model.

Another scenario where random effects are not justified is where marginal, rather than conditional, estimates are needed, in a GLMM context. In this case, generalised estimating equations (GEE) are appropriate instead.

Also, I would also say that random slopes require some care. There has been an explosion in the use of "maximal" models where random slopes are fitted for all the fixed effects. This often leads to overfitting and a singular model. This site contains many questions where researchers have encountered singular models due to over-fitting in the random structure. Some more details about this are here:
Should I remove correlations between random effects before removing some of them?

This last point is quite an interesting one, because we can often reason that ALL fixed effects might vary by subject - in medical research for example, why would we expect a subject's response to anything be exactly the same as all the other subjects ? Obviously it's reasonable to expect them to vary, so why not fit random slopes for everything ? The reason is that there is often insufficient data to support this, and that's what leads to singular models. More info on this is here:
Dealing with singular fit in mixed models
How to simplify a singular random structure when reported correlations are not near +1/-1

Finally there are some domains where researchers are extremely wary of random effects altogether. Econometrics worry about the "higher level exogeneity" which is to do with the assumption of normally distributed random effects and the possible correlation between random effects and residual variation. In my own work I have not found this to be a big problem. It is also an artifact of software - many popular packages for fitting multilevel and mixed effects models make these assumptions for computational efficiency. But if we adopt a Bayesian approach I believe these issues can be (at least in part) overcome.