I work with multiple imputed data and want to compare two proportions of successes/failures using a z-test (one-tailed). Unfortunately, I don't know how to poole the results.
I created 50 complete (imputed) datasets and ran the comparisons resulting in 50 z-scores and p-values. I suppose that I cannot simply average these results but rather have to take the uncertainty of the imputation procedure into account.
What do I have to do to poole my results correctly?
I collected some resources and decided to stick with the following procedure - maybe this helps somebody in the future. As I am still not confident with my solution, I would appreciate any comments and/or advise.
Some background: I conducted a study with two groups, one control and one experimental group. My goal was to compare the proportions of successes of both groups, that is, I analysed the difference between the two proportions. In my control group, out of 60 people 21 were successful, 30 were not successful and 9 were missing (ignoring the missing, 41% were successful). In my experimental group, out of 55 people 27 were successful and 8 were missing (ignoring the missing, 57% successful).
At first, I wanted to do 50 z-tests in the 50 completed data sets but I realised, that with multiple imputation I won't perform 50 hypothesis tests (that's what I asked about in my question) but rather calculate the differences in proportions 50 times and afterwards test this difference. Here is how I did it:
I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.
The variance for the difference in proportions in a single completed data set $i$ is calculated as
$$U_i = (p_{i} * (1 - p_{i}))* (\frac{1}{n_1} + \frac{1}{n_2})$$
($p_i$ being the 50 proportions of successes in the 50 data sets, regardless of the group; $n_1$ and $n_2$ are the group sizes)
I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.008689644$$
I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001116486$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.
I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.00982846$$
The relative increase in variance was calculated as
$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.1310543$$
$$v = (m - 1)(1 + \frac{1}{r})^2 = 3649.726$$
$$\frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - 0.1687879)^2}{0.003769771} = 2.974081$$
$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.
The test statistic is referred to an $F$-distribution with $1$ and $v$ degrees of freedom (Schafer, 1997), resulting in a $p$-value of $0.08469283$.
For reference: The $p$-value of my complete case analysis ($n = 98$) was $~ 0.054$.
That's as far as I got. Here's the literature I used:
Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: Current practice and guidelines. BMC Medical Research Methodology, 9-57.
Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons.
Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data. CRC Press.
