1
$\begingroup$

OP edited to correct for Wolfgang's comments...

I am conducting a meta-analysis of outcome data and am using the metafor package with the recommended logit transformation see related post. My question is about the output. It feels to me like the weights are really off and I don't know what I'm misunderstanding.

For, example, in the following data file (with study source, ni, xi and then the up percentages for illustration only), it's easy to see that the bulk of the results suggest success in less than 50% of the cases because Studies 5-7 have such large sample size.

Study   ni  xi  up
1       8   7   87.5
1       10  10  100
1       11  11  100
2       12  10  83.3
3       13  12  92.3
4       14  12  85.7
4       14  14  100
5       132 55  41.3
6       141 63  44.68
6       141 57  40.43
7       220 80  36.2
7       220 84  38

I used the following commands in metafor

selection2 <- escalc(measure="PLO", xi=xi, ni = ni, data=selection2)
overall2 <- rma.mv(yi, vi, random = ~1 | Study/Group, data = selection2)
predict(overall2, transf=transf.ilogit)

The output for the second part is

    pred  ci.lb  ci.ub  cr.lb  cr.ub <br>
  0.5532   0.3767   1.4685   0.1420  -0.1852   1.2916

I take this to mean that the data suggest an overall success rate of 55%, even though not one of the larger studies shows that effect. Intuitively, this seems to not match my actual data.

Have I made an error in analysis? In interpretation?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ First, take a look at: metafor-project.org/doku.php/… You should add random effects for Study and for each row in the dataset. Also, the output in your post does not match these data. $\endgroup$
    – Wolfgang
    Mar 14, 2017 at 19:43
  • 1
    $\begingroup$ It is perhaps worth pointing out (to the OP, I know @Wolfgang knows) that if you use random effects models the weights tend to become more equal as heterogeneity increases. $\endgroup$
    – mdewey
    Mar 15, 2017 at 9:41
  • $\begingroup$ Tks to both of you. But doesn't it seem strange to use a method that ignores the weights when heterogeneity is present? The method ignores useful information (the largest studies find a fairly consistent effect). Also, is there a reason we don't use sample size as the weights instead of the inverse variance in the case of proportions? The inverse variance is not an independent noise estimate given that it's just part of the transformation of a proportion (whereas the standard deviation is independent of the mean). $\endgroup$
    – jpgerber
    Mar 16, 2017 at 2:14
  • 1
    $\begingroup$ When you do fixed effects you assume a single underlying value so any variation is just sampling variation. When you do random effects you are saying there is no true single underlying value so the variability between the studies is informative and is taken into account in the weighting $\endgroup$
    – mdewey
    Mar 16, 2017 at 15:09
  • $\begingroup$ This aside, I would suggest using a mixed-effects logistic regression model for these data. You have three studies where p=1 and several other studies where p is close to 1. Normal approximations are probably not appropriate here. Use rma.glmm(measure="PLO", xi=xi, ni=ni, data=selection2). $\endgroup$
    – Wolfgang
    Mar 16, 2017 at 19:23

1 Answer 1

Reset to default
1
$\begingroup$

So this does not go unanswered this brings together comments by Wolfgang and me.

The issue is that when there is heterogeneity and a random effects model is used them the weights become more equal. The fixed effects model estimates the hypothesised underlying effect whereas the random effects model estimates the mean and variance of the distribution of true effects.

Note also that the assumption of normality is unlikely to be justified here and as Wolfgang suggested a more appropriate model would be a mixed effects logistic one.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.