13
$\begingroup$

Is precision-based weighting central to meta-analysis? Borenstein et al. (2009) write that for meta-analysis to be possible all that is necessary is that:

  1. Studies report a point estimate which can be expressed as a single number.
  2. Variance can be computed for that point estimate.

It's not immediately clear to me why (2) is strictly necessary. But, indeed, all of the widely accepted methods of meta-analysis rely on precision-based (i.e. inverse variance) weighting schemes, which do require an estimate of variance for each study's effect size. Note that while Hedges' Method (Hedges & Olkin, 1985; Hedges & Vevea, 1998) and Hunter and Schmidt's Method (Hunter & Schmidt, 2004) both basically use sample size weighting, these methods apply only to normalized mean differences, and thus require a standard deviation elsewhere. It makes sense that weights inversely proportional to the variance in each study will minimize the variance in the overall effect size estimator, so is this weighting scheme a requisite feature of all methods?

Is it possible to conduct a systematic review without access to the variance for each effect size and still call the result a meta-analysis? Sample size would seem to have potential as a proxy for precision when variance is unavailable. Could one, for instance, use sample-size weighting in a study where effect size was defined as raw mean difference? How would that effect the consistency and efficiency of the resulting mean effect size?

$\endgroup$
2
  • $\begingroup$ a good question. If you raise a clear specific querry, may help the reader. $\endgroup$
    – user10619
    Commented Jun 7, 2015 at 11:38
  • $\begingroup$ Specifically, I am looking for a way to incorporate a measure of between-study heterogeneity (i.e. random effects) in my weighting scheme when variance data is not available for all studies. An earlier, more specific version of my question can be found here: stats.stackexchange.com/questions/155063/… $\endgroup$
    – Jennifer
    Commented Jun 8, 2015 at 12:42

2 Answers 2

13
$\begingroup$

The question is difficult to answer, because it is so indicative of a general confusion and muddled state-of-affairs in much of the meta-analytic literature (the OP is not to blame here -- it's the literature and the description of the methods, models, and assumptions that is often a mess).

But to make a long story short: No, if you want to combine a bunch of estimates (that quantify some sort of effect, a degree of association, or some other outcome deemed to be relevant) and it is sensible to combine those numbers, then you could just take their (unweighted) average and that would be perfectly fine. Nothing wrong with that and under the models we typically assume when we conduct a meta-analysis, this even gives you an unbiased estimate (assuming that the estimates themselves are unbiased). So, no, you don't need the sampling variances to combine the estimates.

So why is inverse-variance weighting almost synonymous with actually doing a meta-analysis? This has to do with the general idea that we attach more credibility to large studies (with smaller sampling variances) than smaller studies (with larger sampling variances). In fact, under the assumptions of the usual models, using inverse-variance weighting leads to the uniformly minimum variance unbiased estimator (UMVUE) -- well, kind of, again assuming unbiased estimates and ignoring the fact that the sampling variances are actually often not exactly know, but are estimated themselves and in random-effects models, we must also estimate the variance component for heterogeneity, but then we just treated it as a known constant, which isn't quite right either ... but yes, we kind of get the UMVUE if we use inverse-variance weighting if we just squint our eyes very hard and ignore some of these issues.

So, it's efficiency of the estimator that is at stake here, not the unbiasedness itself. But even an unweighted average will often not be a whole lot less efficient than using an inverse-variance weighted average, especially in random-effects models and when the amount of heterogeneity is large (in which case the usual weighting scheme leads to almost uniform weights anyway!). But even in fixed-effects models or with little heterogeneity, the difference often isn't overwhelming.

And as you mention, one can also easily consider other weighting schemes, such as weighting by sample size or some function thereof, but again this is just an attempt to get something close to the inverse-variance weights (since the sampling variances are, to a large extent, determined by the sample size of a study).

But really, one can and should 'decouple' the issue of weights and variances altogether. They are really two separate pieces that one has to think about. But that's just not how things are typically presented in the literature.

However, the point here is that you really need to think about both. Yes, you can take an unweighted average as your combined estimate and that would, in essence, be a meta-analysis, but once you want to start doing inferences based on that combined estimate (e.g., conduct a hypothesis test, construct a confidence interval), you need to know the sampling variances (and the amount of heterogeneity). Think about it this way: If you combine a bunch of small (and/or very heterogeneous) studies, your point estimate is going to be a whole lot less precise than if you combine the same number of very large (and/or homogeneous) studies -- regardless of how you weighted your estimates when calculating the combined value.

Actually, there are even some ways around not knowing the sampling variances (and amount of heterogeneity) when we start doing inferential statistics. One can consider methods based on resampling (e.g., bootstrapping, permutation testing) or methods that yield consistent standard errors for the combined estimate even when we misspecify parts of the model -- but how well these approaches may work needs to be carefully evaluated on a case-by-case basis.

$\endgroup$
0
$\begingroup$

If you know some of the standard errors but not all of them, here is a solution:

(1) assume the unknown SE is drawn randomly from the same distribution as the known SE's or let the distribution of the SE of the estimates of the papers with unknown SE be a free variable. If you want to be fancy, you can use model averaging over these options.

(2) estimate via maximum likelihood

If your study with unknown SE is an 'outlier' the model will explain the anomaly in a combination of these ways:

(a) the study likely had a high SE for its estimate (the study likely has low power)

(b) the study likely has a large random effect component (the researcher picked a data set or method etc. which gives an atypical result)

In effect, this model will reduce the effective precision of the estimate with unknown SE as it becomes more anomalous. In this respect it is highly robust to the inclusion of 'outliers'. At the same time, if you add lots of studies with unknown variance but with results that are typical, the SE or your final estimate will fall.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.