I often hear from my classmates, and even in resources on the internet such as in the abstract to this paper, what I now believe to be a misconception regarding the motivation of using REML (Restricted Maximum Likelihood) to estimate variance components in a random effects model. I want to check with the community that my understanding is correct.

People often start by saying that the "ANOVA Method" (when we use Expected Mean Squares to construct the point estimate of the variance component) can produce negative estimates, which is true. But then they proceed to say that the purpose of REML is to avoid this. I don't think this is strictly correct.

For example, in the model $Y_{ij} = \mu_i + \epsilon_{ij}, \mu_i \sim^{iid} N(\mu,\sigma^2 _\mu ), \epsilon_{ij} \sim^{iid}N(0,\sigma^2), $ with $\mu_i$ and $\epsilon_{ij}$ independent, the REML estimate can be shown to be:

$$\hat{\sigma}^2_\mu =\frac{1}{n}\big[ MS_{Trt} - MSE \big] $$

which can clearly be negative when $MSE > MS_{Trt}$.

In general, the maximum of the restricted likelihood can occur at a point when the estimate of the variance component is negative.

Instead, the purpose of REML is to achieve a less-biased estimate of the variance component as compared to ML. And further, likelihood methods are used in general over the ANOVA method since the ANOVA method only works well in the balanced case; in the unbalanced case, the ANOVA decomposition into sums of squares is not unique so neither is the estimate. (Most of the discussion in this paragraph can be found in Faraway's "Extending the Linear Model with R").

Thanks in advance!

  • $\begingroup$ What is your question? $\endgroup$ – Tim Jul 20 '16 at 19:41
  • $\begingroup$ Thanks for asking. My question is, simply put, "is everything I wrote above correct? And is what I call a 'misconception' incorrect?" Since I have professors, colleagues, and online resources suggesting the misconception I state above, I am seeking confirmation about what is correct. Does that make sense? $\endgroup$ – RMurphy Jul 20 '16 at 20:02
  • $\begingroup$ REML can and will produce negative estimates for variance components if the likelihood surface says to go in that direction. See Searl, Casella & McCulloch. Only the sum is implicitly required to be positive. $\endgroup$ – Nicholas Mancuso Sep 27 '17 at 21:42

It is true that with balanced data the "unconstrained" REML method produces the same estimates as the ANOVA estimates.

But in general, REML is used as an iterative maximization procedure with the non-negativity constraint. For example, both the SAS PROC MIXED and R nlme package REML are based on Lindstrom (1988) which states in the abstract:

[...] propose improvements to the algorithm discussed by Jennrich and Schluchter (1986) to speed convergence and ensure a positive-definite covariance matrix for the random effects at each iteration.

Hence the likelihood optima is searched within the positive variance space.

An fairly exhaustive analysis of the ANOVA method, negative variance and REML you can find in Hocking (2003), Ch 12-14.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.