I am trying to do a metanalysis (using Wilson meta-analysis http://mason.gmu.edu/~dwilsonb/ma.html) comparing the dropout percentage of a treatment with a control group. I tried to calculate the cohen's d using binary proportions. Specifically, I calculated it from this site http://cebcp.org/practical-meta-analysis-effect-size-calculator/standardized-mean-difference-d/binary-proportions/.

The problem is that when using a treatment or control of 100% or 0% the results for probit are NaNs (not a number) or Infinite. An example would be 0/16 and 2/18. Since the values are calculated from log(0)=inf and log(inf)=NaN I considered using extreme values.

The other cohen's d values ranges from -0.2 to +1.14. I have tried to set the NaN/Inifite values fairly high such as 3, 5, 10 and 100 (and for the negatives the coresponding negative values). I have noticed that with 3 some values had non-significant results and as I increased it more values became significant. This seems to be an unreliable method. What choice of values would you suggest? Should I use the already existing extremes? Should I treat them as missing values? Any suggestions or references would be appreciated.

Please note that my background in statistics is not very strong. I am a psychologist.

  • $\begingroup$ Why do you want to convert to Cohen's d? Why not use a method which uses the proportions directly using the log odds ratio? $\endgroup$ – mdewey May 24 '16 at 20:04
  • $\begingroup$ According to my understanding of the book Applied Social Research Methods Mark W. Lipsey, David Wilson-Practical Meta-Analysis-SAGE Publications (2000), that should be the appropriate method as the variable of interest is dichotomous and similarly measured in all studies. Using odds ratio also gives infinite and NaNs for 0%. $\endgroup$ – Pinelopi K May 24 '16 at 20:17

The issue of what to do with zeroes in binary data has received extensive attention in the literature. As you say the odds ratios for the individual studies will be zero or infinite but that can be overcome.

   author = {Bradburn, M J and Deeks, J J and Berlin, J A and Localio, A R},
   title = {Much ado about nothing: a comparison of the performance of
      meta--analytical methods with rare events},
   journal = {Statistics in Medicine},
   year = {2007},
   volume = {26},
   pages = {53--77},
   keywords = {meta-analysis, fixed effects, random effects, sparse data}

provides an overview of a number of methods for dealing with sparse data.

One solution has been to add a small continuity correction to the cells and

   author = {Sweeting, M J and Sutton, A J and Lambert, P C},
   title = {What to add to nothing? {Use} and avoidance of
      continuity corrections in meta--analysis of sparse data},
   journal = {Statistics in Medicine},
   year = {2004},
   volume = {23},
   pages = {1351--1375},
   keywords = {meta-analysis, trials, sparse data}

discuss this in some detail.

   author = {R\"ucker, G and Schwarzer, G and Carpenter, J and Olkin, I},
   title = {Why add anything to nothing? {The} arcsine difference as a
      measure of treatment effect in meta--analysis with zero cells},
   journal = {Statistics in Medicine},
   year = {2009},
   volume = {28},
   pages = {721--738},
   keywords = {meta-analysis, trials, sparse data}

suggest we should avoid the whole problem.

I would suggest starting with the Bradburn reference as it provides advice tailored to different situations and without a bit more knowledge of your precise dataset and scientific question it might be misleading to give you a best buy.

  • $\begingroup$ Thank you for your advice. I was mainly focusing on calculating cohen's d instead of odds ratio. I will look through your references. Do you maybe have any other advice for that? $\endgroup$ – Pinelopi K May 25 '16 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.