I work with multiple imputed data and want to compare two proportions of successes/failures using a z-test (one-tailed). Unfortunately, I don't know how to poole the results.

I created 50 complete (imputed) datasets and ran the comparisons resulting in 50 z-scores and p-values. I suppose that I cannot simply average these results but rather have to take the uncertainty of the imputation procedure into account.

What do I have to do to poole my results correctly?

  • $\begingroup$ If you have an estimate of the effect size (difference in proportions) and its standard error then you would use Rubin's rules to combine them. Is there any particular reason which leads you to prefer combining the $p$-values? $\endgroup$
    – mdewey
    Commented Oct 12, 2016 at 15:11
  • $\begingroup$ Thank you for your reply. I actually do not want to combine the $p$-values but combine the effect sizes and standard errors as you suggested. Unfortunately, I am not sure how to do so. I found this case stats.stackexchange.com/questions/78479/… - does the answer apply to my case as well? I looked the procedure up in the original source from 1987 and citing Rubin it says that "More work is needed on this problem" (p. 102). Is this nevertheless still the way to proceed here? $\endgroup$ Commented Oct 13, 2016 at 9:55
  • $\begingroup$ As far as I know they are still the way to go. There is a site missingdata.lshtm.ac.uk dedicated to missing data which has many esources. Perhaps one of them will help you. $\endgroup$
    – mdewey
    Commented Oct 13, 2016 at 11:58
  • $\begingroup$ Thank you very much for your help. I found a way to get to a test statistic for the combined results. Maybe someone can approve the approach above or give me a hint for a better solution. $\endgroup$ Commented Oct 13, 2016 at 16:21
  • $\begingroup$ Why not post that as an answer? You are allowed to answer your own question. Then people can comment on the answer and vote on it. $\endgroup$
    – mdewey
    Commented Oct 13, 2016 at 17:03

1 Answer 1


I collected some resources and decided to stick with the following procedure - maybe this helps somebody in the future. As I am still not confident with my solution, I would appreciate any comments and/or advise.

Some background: I conducted a study with two groups, one control and one experimental group. My goal was to compare the proportions of successes of both groups, that is, I analysed the difference between the two proportions. In my control group, out of 60 people 21 were successful, 30 were not successful and 9 were missing (ignoring the missing, 41% were successful). In my experimental group, out of 55 people 27 were successful and 8 were missing (ignoring the missing, 57% successful).

At first, I wanted to do 50 z-tests in the 50 completed data sets but I realised, that with multiple imputation I won't perform 50 hypothesis tests (that's what I asked about in my question) but rather calculate the differences in proportions 50 times and afterwards test this difference. Here is how I did it:

  1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

  2. The variance for the difference in proportions in a single completed data set $i$ is calculated as

$$U_i = (p_{i} * (1 - p_{i}))* (\frac{1}{n_1} + \frac{1}{n_2})$$

($p_i$ being the 50 proportions of successes in the 50 data sets, regardless of the group; $n_1$ and $n_2$ are the group sizes)

  1. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.008689644$$

  2. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001116486$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

  3. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.00982846$$

  4. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.1310543$$

  1. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 3649.726$$

  1. Finally, the test-statistic was calculated as

$$\frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - 0.1687879)^2}{0.003769771} = 2.974081$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

The test statistic is referred to an $F$-distribution with $1$ and $v$ degrees of freedom (Schafer, 1997), resulting in a $p$-value of $0.08469283$.

For reference: The $p$-value of my complete case analysis ($n = 98$) was $~ 0.054$.

That's as far as I got. Here's the literature I used:

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: Current practice and guidelines. BMC Medical Research Methodology, 9-57.

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons.

Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data. CRC Press.

  • $\begingroup$ It looks OK except that I think you may have missed out an exponent in step 6 where I think $(1 + \frac{1}{r})$ should be squared. $\endgroup$
    – mdewey
    Commented Oct 14, 2016 at 15:11
  • $\begingroup$ Yes, you are correct, thank you! I actually squared the term but missed it here. $\endgroup$ Commented Oct 14, 2016 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.