Completely different inferences between single-trial difference and meta-analysis comparison

Question

I have the following data

   mydat <- read.table(textConnection('
   study treatment exposure responders
       A         5      277         22
       B         1     1251        211
       B         5      625          8
       C         5      477         36
       C         7      236          6'), header=TRUE)
       

Each (study, treatment) combination produces a risk of responders/exposure. In study B, we see that the risk of Treatment 1 = 211/1251 = 0.169 and Treatment 5 = 8/625 = 0.013, and this risk is significantly different based on a two-sample t-test for just Study B.

Trials A and C reveal risks for Treatment 5 to be 22/277 = 0.079 and 36/477 = 0.075 respectively, still far lower than the Treatment 1 in Study B (although I know, it's from a different trial so results aren't transferable). In any case, a direct comparison in Study B of Treatment 5 vs 1 shows significance, and additional trials still have far lower proportions, so it seems most likely that a network meta-analysis should reveal significance as well. Let's proceed with the network meta-analysis with package gemtc

   require(gemtc)
   net = mtc.network(mydat, treatments = data.frame(id = c(1, 5, 7), description = c("T1", "T2", "T3")))
   model <- mtc.model(net, likelihood = "poisson", link = "log")
   res <- mtc.run(model, n.iter = 100000, thin = 5)
   leaguetab <- data.frame(round(exp(relative.effect.table(res)),2))
   
   leaguetab
                        X1                 X5                X7
   1                     1     0.07 (0, 1.81)    0.02 (0, 2.29)
   5  13.83 (0.55, 346.05)                  5 0.32 (0.01, 8.45)
   7 44.08 (0.44, 4544.36) 3.15 (0.12, 86.25)                 7

The risk ratio of Treatment 5 vs Treatment 1 is 0.07, but the confidence intervals don't show significance. Why is this so? I understand network meta-analysis handles trial hetereogeneity via random effects, but it still seems bizarre it's not significant, since a head-on comparison from Study B reveals significance. All MCMC diagnostics reveal no problems with the sampling either.

Answer 1

The model you are using is a random-effects model that allows for heterogeneity. But with two studies included in the analysis (study A is not even included, since it only has a single arm), there is essentially no information to estimate the amount of heterogeneity. So the heterogeneity prior essentially determines the posterior distribution for the heterogeneity parameter.

By default, mtc.hy.prior("std.dev", "dunif", 0, "om.scale") is used for the prior, so a uniform distribution on $\tau$ where the upper bound for the uniform distribution is determined based on the data (here model$om.scale is equal to around 2.5). The mean/median of a $U(0,2.5)$ distribution is 1.25. Now look at the output from summary(res) and you will find that the posterior mean/median (sd.d) is also 1.25. Also, all the other quantiles given for the posterior match up to what you would expect for such a uniform distribution (e.g., punif(0.0644, 0, 2.5) for the 2.5% quantile is about .025). So the posterior is essentially just the prior.

A mean/median amount of heterogeneity equal to 1.25 according to the posterior is a huge amount of heterogeneity for log incidence rates. Try fitting the model with a more restricted uniform prior:

model <- mtc.model(net, likelihood = "poisson", link = "log", hy.prior=mtc.hy.prior("std.dev", "dunif", 0, 1))

Now the CI for treatment 5 versus 1 is 0.07 (0.02, 0.29) so 1 is not included in the CI and hence the difference is significant.