6
$\begingroup$

I am reading this article which says that "when heterogeneity is present, a confidence interval around the random-effects summary estimate is wider than a confidence interval around a fixed-effect summary estimate."

This is with regards to a meta analysis approach where the outcome is a particular study's overall effect (e.g. mean treatment effect). A fixed effect model will just be a regression model that assumes differences between studies can be attributed to random error $\epsilon$. A random effects model will assume each study has a somewhat different effect, which is mean of all the study effects plus a realization from some random effect distribution.

What I am really confused about is why the random effects model will give larger standard errors to, for example, study level variables used in the meta analysis regression than a fixed effect model will. I know this can happen in random effects models, but often one of the reason for using random effects models is to gain power to detect effects.

Can anyone explain what is going on here?

Edit: To add to the discussion in the comments, here is an example in R of a case where adding a random intercept decreases the standard error of the fixed effect, Age.

library(lme4)
data(Orthodont,package="nlme")
fixed=lm(distance~age,data=Orthodont)
random=lmer(distance ~ age + (1|Subject), data=Orthodont)
summary(fixed)
summary(random)

In both cases, the fixed effect estimate is 0.66, but in the fixed effect model, the standard error for age is 0.1092, while it is 0.06161 in the random effect model.

$\endgroup$
13
  • $\begingroup$ In your question would you like to assume that, to do the meta analysis, you have access only to the summary statistics from multiple studies with their standard error estimates & sample size info? Or you assume that you have the original patient measurements from all the studies? $\endgroup$
    – dipetkov
    Apr 23, 2023 at 11:49
  • $\begingroup$ @dipetkov the former is what I had in mind. $\endgroup$
    – user352188
    Apr 23, 2023 at 12:32
  • $\begingroup$ Thanks for the clarification; you might want to edit your question to highlight this. Furthermore, the claims "the random effects model will give larger standard errors" and "we use random effects models to gain power" contradict each other. The fixed effects analysis should be more powerful; the question would be whether it's justified. See more here: Justifications for a fixed-effects vs random-effects model in meta-analysis. $\endgroup$
    – dipetkov
    Apr 23, 2023 at 12:34
  • $\begingroup$ @dipetkov the contradiction is what my question is about; something I have seen implied many times is that random effects models will lower the standard errors of the estimates compared to a fixed effects model. I am aware that this is not ALWAYS the case, but to read now that with respect to meta analysis, the standard errors will always INCREASE when using random effects versus fixed effects is extremely surprising as I've never heard that said about random effects models outside of meta analysis. As far as I'm concerned a meta regression is no different from any other regression? $\endgroup$
    – user352188
    Apr 23, 2023 at 12:41
  • $\begingroup$ I agree that always is not a word that accords well with the practice of statistics. Then on the other hand, "I've seen implied many times" is not exactly convincing either. Do you have any references at hand? $\endgroup$
    – dipetkov
    Apr 23, 2023 at 13:12

2 Answers 2

1
$\begingroup$

The standard regression without random effect assumes that all observations in all studies can be used to estimate the same main effect. The effective sample size for estimating this main effect then is the number of observations overall.

In a model with a random effect, there is an underlying study mean, which is the main effect plus the random deviation of the specific study from the main effect. Each observation contributes to estimating what I call the underlying study mean, so observations in different studies estimate different things. The overall main effect can only be estimated bringing the different study effects together. So the effective sample size would be the number of studies (in fact less than this because a study with a low number of observations will estimate it's own study mean potentially very imprecisely), which is of course lower than the number of observations overall. This means that the estimation of the main effect will be much less precise under the model involving random study effects.

Be wary though of taking this as an argument for using the model without random effect, as this model may not capture the realistic uncertainty appropriately, so the larger precision you get may well be deceptive.

It is not appropriate to compare the behaviour of a standard regression assuming that no random effect exists with a model with random effect assuming that a random effect exists. Obviously each model will do better in a situation in which its model assumptions are fulfilled, but the random effects model is more general, i.e., even if the random effect variance is in fact zero, i.e., the random effect does not exist, the random effect model will do an OK job, whereas the standard regression may well not, in case a random effect exists.

If I understand things correctly, this by the way may play out both in terms of the test level, i.e., the model without random effect may reject a null model too easily in case a random effect exists, and in terms of power, or in only one of these respects, probably depending on the number of studies, and how the within-study variance relates to the variance between studies. (My intuition is that anticonservativity, i.e., too large rejection probability under the null hypothesis, will likely be the dominating issue here, rather than potential loss of power.)

Fun fact: If a single study is run, this will never be modelled involving a random study effect, as such effect cannot be identified based on one study only. This means that if we consider a single study, the effective sample size will be the number of observations, say, 200. Now let's say three further studies are run, and somebody meta-analyses these using a random effect. This additional information reduces the effective sample size to 4, the number of studies! We become far less precise by observing more.

The explanation is that any potential problem causing the first study to in fact deviate on average from the overall main effect cannot be detected as long as further studies do not exist, i.e., we rely on a model that makes stronger assumptions not because there is any more reason that this is true, but rather because this is our only chance to say anything.

PS: I add something after having seen the added Orthodont data example in the question. In fact two things happen when adding a random effect. One is the change of the effective sample size, already discussed above. The other one is that the random effect will explain some of the variation in the data, and for this reason the fixed effect will account for less variation and can in principle be estimated more precisely. Both of these point in opposite directions, so the estimated precision of the fixed effect estimators may go up as well as down when adding a random effect. Factors for having an estimated standard error of the fixed estimator rather larger when adding a random effect are (1) the number of clusters is very small compared to the number of observations within clusters (as often happens in meta analysis) and (2) the variance of the random effect is rather low, compared to the within-clusters variance.

$\endgroup$
10
  • $\begingroup$ Thanks for your answer, and I get the point about going from the totality of the data points from one study to 4 data points when comparing 4 studies, but it's hardly ever the case that individual level data is available for any sizeable number of studies. I'm thinking about the case where you have 50 studies, each with one simple result and variance, and you treat each one as a data point in a fixed effect regression analysis. Moving from that model to one with random effects apparently increases the standard errors. But there are many examples outside meta analysis where the opposite happens $\endgroup$
    – user352188
    Apr 23, 2023 at 12:53
  • $\begingroup$ I'm wondering maybe if it's because although you are using random effects there is no CLUSTERING in the data (or each study is a cluster of 1) whereas in e.g. a longitudinal study, you get multiple data points from each subject (which each has their own random effect -- the "study" from the meta analysis is equivalent to the subject here) ? To me it does make some sense that clusters of size 1 would detract from the power 100% of the time instead of going one or the other way depending on the data in hand. @dipetkov $\endgroup$
    – user352188
    Apr 23, 2023 at 12:58
  • $\begingroup$ @user352188 My understanding (of your question as well as of standard practices) is that if you fit a standard regression without random effect, you do something that is equivalent to using every single observation in any study as one observation. Even though you technically may run a regression where every study is a "data point", these data points will be implicitly weighted by their variances, amounting to an overall regression on all observations. I hope we are on the same page here... $\endgroup$ Apr 23, 2023 at 13:05
  • 1
    $\begingroup$ @user352188 The studies are the clusters. And each study has (hopefully) many observations. $\endgroup$ Apr 23, 2023 at 13:11
  • 1
    $\begingroup$ @user352188 when we are using a paired t-test in place of an unpaired t-test, then we are adding a fixed effect (for each pair), and not a random effect. The potential improvement is due to the additional effect, and not due to replacing a fixed effect by a random effect. A paired t-test is not a random effects model. $\endgroup$ Apr 23, 2023 at 13:53
1
$\begingroup$

If studies relate to a random effect between different studies, then the 'effective degrees of freedom' will be different, and also relate to the number of studies asside from the sum of sample sizes.

It can be illustrated intuitively with an extreme case where the random effect of the study is large. Imagine you have three studies with each 100 observations.

example of studies with large differences

The mean of the points is around -2.087, and if you assume that all those point come from the same normal distribution, then the standard error is ~0.143 based on the sample size of 300. But, these three samples are obviously not from a single normal distribution, and the effective sample size looks more like 3 (the number of studies) and the standard error should be ~1.55.


The studies above might be represented in a table like:

study id    mean    standard error
1           -3.025  0.090
2            0.932  0.098
3           -4.167  0.104

And the mean effect could be estimated like

$$\hat{m} = \frac{-3.025+0.932-4.167}{3} = -2.087$$

and the standard error is (wrongly) computed as combining the standard errors of the different studies

$$SE(\hat{m}) = \sqrt{\frac{0.90^2+0.098^2+0.104^2}{9}} = 0.056$$

This relates to the question: Statistics question: Why is the standard error, which is calculated from 1 sample, a good approximation for the spread of many hypothetical means? We can use the variance within a sample to estimate the standard error of the estimate of the population mean from which the sample is taken. However, the standard errors of the studies are a measure for the variance within the individual studies and not for the variance between the studies.


I know this can happen in random effects models, but often one of the reason for using random effects models is to gain power to detect effects.

I am not sure how random effects can increase power in comparison to fixed effects. But possibly this relates to the introduction of interaction terms that reduce the degrees of freedom on the one hand, but can improve the model fit and reduce the residuals on the other hand. In this case the improvement in power relates not to the cases 'random effect' versus 'fixed effect' bit to 'additional effect' versus 'no additional effect'.

$\endgroup$
12
  • 1
    $\begingroup$ At least to me, this answer is not clear on the difference between meta analysis and mixed models (with a random study effect). In either case 3 studies is probably not enough to treat their effect as random. $\endgroup$
    – dipetkov
    Apr 23, 2023 at 12:08
  • $\begingroup$ @dipetkov the 3 studies are random because that is how I simulated them. The true effect was 0 in my simulation, but the observation was a mean effect around 2.187, and when we assume no random effects for the studies, and instead assume that all the studies have the same mean, then we would erroneously estimates the standard error of the estimate of that mean to be ~0.143 $\endgroup$ Apr 23, 2023 at 12:20
  • $\begingroup$ I'm not sure how any of this addresses my confusion, which is that if in the real world we had 3 effect estimates from 3 different studies, it might not be wise to model them as random effects. $\endgroup$
    – dipetkov
    Apr 23, 2023 at 12:30
  • $\begingroup$ The contrast between 'fixed effects model' and 'random effects model' is not the issue whether 'the effect of the study' is modelled as fixed versus random. Instead it is whether we should model all studies as a single fixed effect versus adding a random effect on top of that. It is not about replacing a fixed effect by a random effect, but by adding a mixed effect to a random effect. $\endgroup$ Apr 23, 2023 at 12:53
  • $\begingroup$ The word "effect" got overloaded. By "study effect" I mean the summary statistic (say standardized mean difference between treatment and control) which is reported as the result from the study analysis. $\endgroup$
    – dipetkov
    Apr 23, 2023 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.