A colleague of mine is conducting a meta-analysis for which he has available the “raw” mean differences for each study along with the sample sizes. The differences are between-subjects effects, comparing older and younger populations in terms of memory performance as measured by d’ in either population (this is a signal detection measure based on a probit transformation of a difference in the proportions for targets and lures); all data were on the same scale. Estimates of variability were not widely available and are therefore being ignored. Instead, he is conducting a multilevel model using sample sizes as the weights rather than the usual inverse variance. He is including random effects for sample, experiment and article. This has worked out pretty well, as far as I can tell. He would like to test for publication bias using a technique similar to Egger’s Regression test, only using mean difference (rather than a standardized mean difference) as his DV and using some transformation of sample size (rather than precision) as his predictor.

The Question

  1. Could someone point me in the direction of work using sample size as a predictor in a regression test for publication bias? The closest I could find was Peters test – which uses inverse sample size in its calculations, but seems to use a weighting scheme calibrated for log-odds ratios. Is there a modification appropriate for continuous measures?

  2. Are there any other approaches to publication bias that could be fit using a continuous "raw" mean difference and sample size?

  3. As a supplementary question, would it make sense to honour the same random-effects structure in the regression test for publication bias as in the meta-analysis?

  • 2
    $\begingroup$ You put raw in quotes... is really only the d' available or is also available the proportion correct? If you have both of those you can derive the variance. Actually, if you only had proportion correct you might be able to also derive a potential theoretical variance. $\endgroup$ – John May 13 '17 at 17:47
  • $\begingroup$ @John Fancy seeing you here! As it happens, I pointed this out to my colleague already. There are ways to estimate the variance of d' from the proportions. He did not seem interested in doing so (or at least, his follow-up questions still focused on sample size) so I do not expect that is an avenue they wish to pursue. I do know this was a major undertaking – including something like 300 coded studies – so perhaps it would require re-coding they do not want to undertake. I am not sure. In any event, for the time being I have assumed we only have the above. $\endgroup$ – jmfawcet May 13 '17 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.