3
$\begingroup$

I have observed in many meta-analyses, and in tutorials on the matter, that the analyzed studies are commonly weighted by the reciprocal of the standard error. This seems to have a face validity because a lower standard error suggests a better estimate of the effect in question.

However, a standard error could be low for two reasons: (1) it could be because there is a larger sample size $n$ in a study; or (2) it could be because there is a lower variance in the study sample(s). Weighting by $n$ makes sense because one should weight the study not just by the more accurate estimate but also because the study is providing more evidence. And weighting by standard error would tend to be highly correlated with weighting by $n$. But why should low sample variance also matter? It could simply be a very unrepresentative sample in a study that causes the variance to be low. In that case one should not be weighting the study heavily.

Also, when doing a fixed effects meta-analysis it seems to violate the assumptions of the analysis to weight by study standard errors since one would expect there is really only one variance and all studies deviate from that variance by some amount. Once study variances are assumed equal weighting by standard error should be weighting by $n$.

So, why is weighting by standard error so often favoured over weighting by $n$?

$\endgroup$
4
$\begingroup$

I suspect they are not weighting by the standard error, which would mean giving higher weight to less certain results, but weighting by the reciprocal of the square of the standard error, i.e. inverse-variance weighting, because this has the least variance among all weighted averages.

For example the Cochrane Statistical Methods Group suggests the generic inverse-variance weighted average $$\frac{\sum Y_i (1/SE_i^2)}{\sum (1/SE_i^2)}$$ with a reference to Rice K, Higgins JPT, Lumley T. A re-evaluation of fixed effect(s) meta-analysis. Journal of the Royal Statistical Society Series A (Statistics in Society) 2018; 181: 205-227.

Since $SE_i^2$ is an estimator of $\frac1{n_i}\sigma_i^2$, you might hope that its reciprocal was approximately proportional to $n_i$ if all the $\sigma_i^2$ are the same and so you might get something similar to your suggestion of weighting by $n_i$. But there are two additional reasons for this particular method:

  • the individual studies may have taken different approaches to estimating the same parameter, some more accurate than others, and this difference in precision can be taken into account
  • most studies may report a standard error for their estimate even when they are more confused about the actual $n$ used (especially if parts of the original sample are dropped for one reason or another)
$\endgroup$
6
  • $\begingroup$ Yes, of course, I meant the reciprocal. I have edited to reflect the intent. While these seem reasonable suggestions it still seems a stretch to believe that differences in sample variance may be more attributable to measurement quality than sampling error. If that's the primary reason it's on pretty shaky ground. $\endgroup$
    – John
    Sep 19 at 1:27
  • 2
    $\begingroup$ @John - suppose you were estimating expected post-treatment survival time. If one study followed $100$ people for a year noting the dates of death of $10$ of them though the others were still alive, and another followed $100$ people for $10$ years noting the dates of death of $65$ though the others were still alive, it is not immediately obvious what the sample sizes were. $\endgroup$
    – Henry
    Sep 19 at 2:02
  • $\begingroup$ "It could simply be a very unrepresentative sample in a study that causes the variance to be low" This is why a risk of bias assessment is usually recommended - otherwise even weighting by $n$ is a bad idea. $\endgroup$ Sep 19 at 2:09
  • $\begingroup$ @Henry ah that makes sense. I had not considered the situations where the SE is calculable but the N isn't clear. I agree that in that case SE is definitely what you want to go with. But it's not really a good justification for SE as a general rule. $\endgroup$
    – John
    Sep 21 at 1:58
  • 1
    $\begingroup$ I didn't follow your wording; I meant that weighting can't solve problems of non-representative (or high risk of bias) samples. If the issue of concern is solely outlying sample variances, and fixed effects is a weak assumption, weighting by sample size makes sense. $\endgroup$ Sep 21 at 5:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.