Fixed vs. random effect meta regression

Question

I am reading this article which says that "when heterogeneity is present, a confidence interval around the random-effects summary estimate is wider than a confidence interval around a fixed-effect summary estimate."

This is with regards to a meta analysis approach where the outcome is a particular study's overall effect (e.g. mean treatment effect). A fixed effect model will just be a regression model that assumes differences between studies can be attributed to random error $\epsilon$. A random effects model will assume each study has a somewhat different effect, which is mean of all the study effects plus a realization from some random effect distribution.

What I am really confused about is why the random effects model will give larger standard errors to, for example, study level variables used in the meta analysis regression than a fixed effect model will. I know this can happen in random effects models, but often one of the reason for using random effects models is to gain power to detect effects.

Can anyone explain what is going on here?

Edit: To add to the discussion in the comments, here is an example in R of a case where adding a random intercept decreases the standard error of the fixed effect, Age.

library(lme4)
data(Orthodont,package="nlme")
fixed=lm(distance~age,data=Orthodont)
random=lmer(distance ~ age + (1|Subject), data=Orthodont)
summary(fixed)
summary(random)

In both cases, the fixed effect estimate is 0.66, but in the fixed effect model, the standard error for age is 0.1092, while it is 0.06161 in the random effect model.

Answer 1

The standard regression without random effect assumes that all observations in all studies can be used to estimate the same main effect. The effective sample size for estimating this main effect then is the number of observations overall.

In a model with a random effect, there is an underlying study mean, which is the main effect plus the random deviation of the specific study from the main effect. Each observation contributes to estimating what I call the underlying study mean, so observations in different studies estimate different things. The overall main effect can only be estimated bringing the different study effects together. So the effective sample size would be the number of studies (in fact less than this because a study with a low number of observations will estimate it's own study mean potentially very imprecisely), which is of course lower than the number of observations overall. This means that the estimation of the main effect will be much less precise under the model involving random study effects.

Be wary though of taking this as an argument for using the model without random effect, as this model may not capture the realistic uncertainty appropriately, so the larger precision you get may well be deceptive.

It is not appropriate to compare the behaviour of a standard regression assuming that no random effect exists with a model with random effect assuming that a random effect exists. Obviously each model will do better in a situation in which its model assumptions are fulfilled, but the random effects model is more general, i.e., even if the random effect variance is in fact zero, i.e., the random effect does not exist, the random effect model will do an OK job, whereas the standard regression may well not, in case a random effect exists.

If I understand things correctly, this by the way may play out both in terms of the test level, i.e., the model without random effect may reject a null model too easily in case a random effect exists, and in terms of power, or in only one of these respects, probably depending on the number of studies, and how the within-study variance relates to the variance between studies. (My intuition is that anticonservativity, i.e., too large rejection probability under the null hypothesis, will likely be the dominating issue here, rather than potential loss of power.)

Fun fact: If a single study is run, this will never be modelled involving a random study effect, as such effect cannot be identified based on one study only. This means that if we consider a single study, the effective sample size will be the number of observations, say, 200. Now let's say three further studies are run, and somebody meta-analyses these using a random effect. This additional information reduces the effective sample size to 4, the number of studies! We become far less precise by observing more.

The explanation is that any potential problem causing the first study to in fact deviate on average from the overall main effect cannot be detected as long as further studies do not exist, i.e., we rely on a model that makes stronger assumptions not because there is any more reason that this is true, but rather because this is our only chance to say anything.

PS: I add something after having seen the added Orthodont data example in the question. In fact two things happen when adding a random effect. One is the change of the effective sample size, already discussed above. The other one is that the random effect will explain some of the variation in the data, and for this reason the fixed effect will account for less variation and can in principle be estimated more precisely. Both of these point in opposite directions, so the estimated precision of the fixed effect estimators may go up as well as down when adding a random effect. Factors for having an estimated standard error of the fixed estimator rather larger when adding a random effect are (1) the number of clusters is very small compared to the number of observations within clusters (as often happens in meta analysis) and (2) the variance of the random effect is rather low, compared to the within-clusters variance.

Answer 2

If studies relate to a random effect between different studies, then the 'effective degrees of freedom' will be different, and also relate to the number of studies asside from the sum of sample sizes.

It can be illustrated intuitively with an extreme case where the random effect of the study is large. Imagine you have three studies with each 100 observations.

The mean of the points is around -2.087, and if you assume that all those point come from the same normal distribution, then the standard error is ~0.143 based on the sample size of 300. But, these three samples are obviously not from a single normal distribution, and the effective sample size looks more like 3 (the number of studies) and the standard error should be ~1.55.

---

The studies above might be represented in a table like:

study id    mean    standard error
1           -3.025  0.090
2            0.932  0.098
3           -4.167  0.104

And the mean effect could be estimated like

$$\hat{m} = \frac{-3.025+0.932-4.167}{3} = -2.087$$

and the standard error is (wrongly) computed as combining the standard errors of the different studies

$$SE(\hat{m}) = \sqrt{\frac{0.90^2+0.098^2+0.104^2}{9}} = 0.056$$

This relates to the question: Statistics question: Why is the standard error, which is calculated from 1 sample, a good approximation for the spread of many hypothetical means? We can use the variance within a sample to estimate the standard error of the estimate of the population mean from which the sample is taken. However, the standard errors of the studies are a measure for the variance within the individual studies and not for the variance between the studies.

---

I know this can happen in random effects models, but often one of the reason for using random effects models is to gain power to detect effects.

I am not sure how random effects can increase power in comparison to fixed effects. But possibly this relates to the introduction of interaction terms that reduce the degrees of freedom on the one hand, but can improve the model fit and reduce the residuals on the other hand. In this case the improvement in power relates not to the cases 'random effect' versus 'fixed effect' bit to 'additional effect' versus 'no additional effect'.