Practical data collection tips for performing a network meta analysis (mixed treatment comparisons)

Question

As described previously in this post, I am currently working through issues with conducting a network meta analysis (also known as mixed treatment comparisons). For those unfamiliar with this method, let's say that you're interested in conducting a meta analysis that compares treatment 1 with treatment 2, treatment 3, and a control condition. However, a typical set of studies might make the following comparisons:

• Study 1: Treatment 1 vs treatment 2
• Study 2: Treatment 1 vs treatment 3
• Study 3: Treatment 2 vs treatment 3
• Study 4: Treatment 1 vs control

Network meta-analysis is a way of aggregating information across these kinds of studies into one analysis. In particular, network meta-analysis makes use of indirect information by using study 1 and study 2 to obtain an estimate of the treatment 2 vs treatment 3 effect, even though treatment 2 and 3 were never directly compared in a head-to-head trial (see, for example, Salanti (2012) for more information).

My question is a practical one about the kinds of information you need from an article in order to use it in a meta analysis. It's probably easiest to understand what this question means by way of analogy to a conventional pairwise meta-analysis.

Let's go back to a situation where one is interested in only comparing treatment 1 to a control in a set of studies. For simplicity, let's assume that the effect size metric of interest is Cohen's d (the difference in group means divided by the pooled standard deviation of the groups). Even if not reported in a given paper, Cohen's d is easily calculated given the means and standard deviations of the treatment and control groups. Sometimes, however, these simple descriptive statistics are not reported; in this case, Cohen's d can still be back-calculated given sufficient information, such as:

1. The observed t value from a t-test, plus the sample sizes of the two groups
2. The observed F value from an ANOVA, plus the sample sizes of the two groups
3. The observed regression coefficient, plus the p-value of its test against 0

And so on. A wide variety of ways to calculate appropriate effect size metrics are described, for example, in Lipsey and Wilson (2000) and in a wide variety of other texts.

Because network meta-analysis is relatively new, however, I haven't found any similar kinds of advice about how to extract the proper summary statistics when they are not reported in a paper. It seems to me that, because network meta-analysis uses indirect information across studies, it requires the actual means and standard deviations; information on inferential statistics seems to me to be insufficient.

Do any of you have any practical advice about extracting information from incomplete research papers for a network meta-analysis? I'm hoping that there's some way of dealing with this incomplete information, as in my retrieved studies for my own meta-analysis, means and standard deviations are seldom reported.

Answer 1

In principle, there is nothing special about computing the required effect sizes for a network meta-analysis. Let's stick to Cohen's d here. So, for each study, you just compute the $d$ value, either using the raw means and SDs or via some appropriate transformation of some other statistic.

As a simple example (what you numbered 1), we can compute $d = t \sqrt{1/n_1 + 1/n_2}$, where $t$ denotes the test statistic of an independent samples t-test comparing the two groups and $n_1$ and $n_2$ the two group sizes. One just has to be careful to get the sign of $d$ right, since authors may have computed the t-statistic with $\bar{x}_1 - \bar{x}_2$ or the other way around (hopefully the text makes it clear which group had the higher mean, so that there is no question about the sign).

Lipsey and Wilson (2000) is indeed a very good reference for the kinds of transformations one can use to obtain those $d$ values based on various pieces of information.

For each d value, you also need to code which two conditions are being compared. This can be done most easily by coding dummy variables (one for each condition) with $+1$ and $-1$. So, for example, for the 4 studies in your question above, your coding would be:

         trt1 trt2 trt3 ctrl
study 1  +1   -1   0    0 
study 2  +1   0    -1   0
study 3  0    +1   -1   0
study 4  +1   0    0    -1

Sometimes you will have studies with more than 2 conditions, for example a study that compared treatments 1 and 2 versus control. Here, you can compute two $d$ values (technically, you could compute 3, but one is redundant), such as treatment 1 versus control and treatment 2 versus control. Again, the computation of those two $d$ values can be done by any means necessary (no pun intended -- well, ok, maybe a slight pun was intended). The coding of the dummy variables above is again easy (you just have two rows for that study). You will also want to have a variable that codes the study factor (so that you know that those two $d$ values came from the same study).

A bit more tricky is the fact that those two $d$ values are no longer statistically independent (since the information from the control group was used twice). So, in addition to computing the sampling variances for those two $d$ values, you also need to compute the covariance. Equations for all of that can be found in Chapter 19 by Gleser and Olkin in the Handbook of Research Synthesis and Meta-Analysis (2nd ed.)

So, in the end, you will have the vector with the $d$ values, the corresponding coding of the contrasts via the dummy variables, and the corresponding variance-covariance matrix of the $d$ values. That variance-covariance matrix is diagonal if you have no studies with more than 2 conditions. If there are studies with more than 2 conditions, then you have a block-diagonal structure.

That's the information you will need for the analysis. More on that can be found in the literature. A good reference would be Salanti et al. (2008).

Answer 2

All the meta-analytic models I have seen require the raw data but if someone is familiar with WinBugs, they might be able to help modify the code to fit your needs. There is a BUGS ListServ that Bayesians frequently post on. Maybe someone there can give you more guidance.

Ahmed