Skip to main content
5 of 6
added 1200 characters in body

I collected some resources and decided to stick with the following procedure - maybe this helps somebody in the future. As I am still not confident with my solution, I would appreciate any comments and/or advise.

Some background: I conducted a study with two groups, one control and one experimental group. My goal was to compare the proportions of successes of both groups, that is, I analysed the difference between the two proportions. In my control group, out of 60 people 21 were successful, 30 were not successful and 9 were missing (ignoring the missing, 41% were successful). In my experimental group, out of 55 people 27 were successful and 8 were missing (ignoring the missing, 57% successful).

At first, I wanted to do 50 z-tests in the 50 completed data sets but I realised, that with multiple imputation I won't perform 50 hypothesis tests (that's what I asked about in my question) but rather calculate the differences in proportions 50 times and afterwards test this difference. Here is how I did it:

  1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

  2. The variance for a proportion in a single completed data set $i$ is calculated as

$$U_i = \frac{p_{i} * (1 - p_{i})}{n}$$

($p_i$ being the 50 proportions of successes in the 50 data sets, regardless of the group; $n$ is the sample size)

  1. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.002167831$$

  2. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001570529$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

  3. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.09480994$$

  4. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.7389596$$

  1. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 271.3524$$

  1. Finally, the test-statistic was calculated as

$$\frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - 0.1687879)^2}{0.003769771} = 7.557316$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

The test statistic is referred to an $F$-distribution with $1$ and $v$ degrees of freedom (Schafer, 1997), resulting in a $p$-value of $0.00637792$.

For reference: The $p$-value of my complete case analysis ($n = 98$) was $~ 0.054$.

That's as far as I got. Here's the literature I used:

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: Current practice and guidelines. BMC Medical Research Methodology, 9-57.

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons.

Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data. CRC Press.