As described previously in this post, I am currently working through issues with conducting a network meta analysis (also known as mixed treatment comparisons). For those unfamiliar with this method, let's say that you're interested in conducting a meta analysis that compares treatment 1 with treatment 2, treatment 3, and a control condition. However, a typical set of studies might make the following comparisons:
- Study 1: Treatment 1 vs treatment 2
- Study 2: Treatment 1 vs treatment 3
- Study 3: Treatment 2 vs treatment 3
- Study 4: Treatment 1 vs control
Network meta-analysis is a way of aggregating information across these kinds of studies into one analysis. In particular, network meta-analysis makes use of indirect information by using study 1 and study 2 to obtain an estimate of the treatment 2 vs treatment 3 effect, even though treatment 2 and 3 were never directly compared in a head-to-head trial (see, for example, Salanti (2012) for more information).
My question is a practical one about the kinds of information you need from an article in order to use it in a meta analysis. It's probably easiest to understand what this question means by way of analogy to a conventional pairwise meta-analysis.
Let's go back to a situation where one is interested in only comparing treatment 1 to a control in a set of studies. For simplicity, let's assume that the effect size metric of interest is Cohen's d (the difference in group means divided by the pooled standard deviation of the groups). Even if not reported in a given paper, Cohen's d is easily calculated given the means and standard deviations of the treatment and control groups. Sometimes, however, these simple descriptive statistics are not reported; in this case, Cohen's d can still be back-calculated given sufficient information, such as:
- The observed t value from a t-test, plus the sample sizes of the two groups
- The observed F value from an ANOVA, plus the sample sizes of the two groups
- The observed regression coefficient, plus the p-value of its test against 0
And so on. A wide variety of ways to calculate appropriate effect size metrics are described, for example, in Lipsey and Wilson (2000) and in a wide variety of other texts.
Because network meta-analysis is relatively new, however, I haven't found any similar kinds of advice about how to extract the proper summary statistics when they are not reported in a paper. It seems to me that, because network meta-analysis uses indirect information across studies, it requires the actual means and standard deviations; information on inferential statistics seems to me to be insufficient.
Do any of you have any practical advice about extracting information from incomplete research papers for a network meta-analysis? I'm hoping that there's some way of dealing with this incomplete information, as in my retrieved studies for my own meta-analysis, means and standard deviations are seldom reported.