Pitfalls of linear mixed models

Question

What are some of the main pitfalls of using linear mixed-effects models? What are the most important things to test/watch out for in assessing the appropriateness of your model? When comparing models of the same dataset, what are the most important things to look for?

Answer 1

This is a good question.

Here are some common pitfalls:

1. Using standard likelihood theory, we may derive a test to compare two nested hypotheses, $H_0$ and $H_1$, by computing the likelihood ratio test statistic. The null distribution of this test statistic is approximately chi-squared with degrees of freedom equal to difference in the dimensions of the two parameters spaces. Unfortunately, this test is only approximate and requires several assumptions. One crucial assumption is that the parameters under the null are not on the boundary of the parameter space. Since we are often interested in testing hypotheses about the random effects that take the form: $$H_0: \sigma^2=0$$ This a real concern. The way to get around this problem is using REML. But still, the p-values will tend to be larger than they should be. This means that if you observe a significant effect using the χ2 approximation, you can be fairly confident that it is actually significant. Small, but not significant, p-values might spur one to use more accurate, but time-consuming, bootstrap methods.

2. Comparing fixed effects: If you plan to use the likelihood ratio test to compare two nested models that differ only in their fixed effects, you cannot use the REML estimation method. The reason is that REML estimates the random effects by considering linear combinations of the data that remove the fixed effects. If these fixed effects are changed, the likelihoods of the two models will not be directly comparable.

3. P-values: The p-values generated by the likelihood ratio test for fixed effects are approximate and unfortunately tend to be too small, thereby sometimes overstating the importance of some effects. We may use nonparametric bootstrap methods to find more accurate p-values for the likelihood ratio test.

4. There are other concerns about p-values for the fixed effects test which are highlighted by Dr. Doug Bates [here].

I am sure other members of the forum will have better answers.

Source: Extending linear models with R -- Dr. Julain Faraway.

Answer 2

The common pitfall which I see is the ignoring the variance of random effects. If it is large compared to residual variance or variance of dependent variable, the fit usually looks nice, but only because random effects account for all the variance. But since the graph of actual vs predicted looks nice you are inclined to think that your model is good.

Everything falls apart when such model is used for predicting new data. Usually then you can use only fixed effects and the fit can be very poor.

Answer 3

Modeling the variance structure is arguably the most powerful and important single feature of mixed models. This extends beyond variance structure to include correlation among observations. Care must be taken to build an appropriate covariance structure otherwise tests of hypotheses, confidence intervals, and estimates of treatment means may not be valid. Often one needs knowledge of the experiment to specify the correct random effects.

SAS for Mixed Models is my go to resource, even if I want to do the analysis in R.