Assessing normality assumption of effect sizes in meta-regression

Question

I'm considering options for estimating tau-squared (the true between-study variance) in meta-regression, and I know method-of-moments (MM), unrestricted maximum likelihood (ML) and restricted maximum likelihood (REML) is commonly used.

As I see it, the choice between MM or ML/REML is mainly based on whether we assume that effect sizes are normally distributed or not. If we are not willing to assume this, MM is preferred and if we are willing to assume this, we can either choose ML or REML.

To test whether effect sizes are normally distributed, I have made a variable containing effect sizes for each study and tested normality graphically with a standardized normal probability plot ("pnorm" in Stata) and quantiles of effect sizes against the normal distribution ("qnorm" in Stata). I have also performed the Shapiro-Wilk test and cannot reject the null hypothesis of a normal distribution.

Based on these considerations, I have chosen ML instead of MM. However, I am unsure of whether the assessment of normal distribution is correct, and I have not found other procedures for this. Thus, input on this and/or references would be highly appreciated.

Answer 1

You might be interested in the article by Viechtbauer entitled "Bias and efficiency of meta--analytic variance estimators in the random--effects model" available here which compares five estimators, the three you mention and also Hunter-Schmidt and Hedges'. No one estimate comes out as superior on both bias and efficiency under all circumstances. The maximum likelihood one which you have chosen has negative bias (as it usually does) so you might prefer to go with REML.

Added as edit

There is also a more recent article "Methods to estimate the between-study variance and its uncertainty in meta-analysis" available here which is open access.