Return to Answer

I collected some resources and decided to stick with the following procedure - maybe this helps somebody in the future. As I am still not confident with my solution, I would appreciate any comments and/or advise.

Some background: I conducted a study with two groups, one control and one experimental group. My goal was to compare the proportions of successes of both groups, that is, I analysed the difference between the two proportions. In my control group, out of 60 people 21 were successful, 30 were not successful and 9 were missing (ignoring the missing, 41% were successful). In my experimental group, out of 55 people 27 were successful and 8 were missing (ignoring the missing, 57% successful).

At first, I wanted to do 50 z-tests in the 50 completed data sets but I realised, that with multiple imputation I won't perform 50 hypothesis tests (that's what I asked about in my question) but rather calculate the differences in proportions 50 times and afterwards test this difference. Here is how I did it:

1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

2. The variance for a proportion in a single completed data set $i$ is calculated as

$$U_i = \frac{p_{i} * (1 - p_{i})}{n}$$

($p_i$ being the 50 proportions of successes in the 50 data sets, regardless of the group; $n$ is the sample size)

3. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.002167831$$

4. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001570529$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

5. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.09480994$$

6. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.7389596$$

6. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 271.3524$$

7. Finally, the test-statistic was calculated as

$$\frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - 0.1687879)^2}{0.003769771} = 7.557316$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

The test statistic is referred to an $F$-distribution with $1$ and $v$ degrees of freedom (Schafer, 1997), resulting in a $p$-value of $0.00637792$.

For reference: The $p$-value of my complete case analysis ($n = 98$) was $~ 0.054$.

That's as far as I got. Here's the literature I used:

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: Current practice and guidelines. BMC Medical Research Methodology, 9-57.

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons.

Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data. CRC Press.

I collected some resources and decided to stick with the following procedure - maybe this helps somebody in the future. As I am still not confident with my solution, I would appreciate any comments and/or advise.

Some background: I conducted a study with two groups, one control and one experimental group. My goal was to compare the proportions of successes of both groups, that is, I analysed the difference between the two proportions. In my control group, out of 60 people 21 were successful, 30 were not successful and 9 were missing (ignoring the missing, 41% were successful). In my experimental group, out of 55 people 27 were successful and 8 were missing (ignoring the missing, 57% successful).

At first, I wanted to do 50 z-tests in the 50 completed data sets but I realised, that with multiple imputation I won't perform 50 hypothesis tests (that's what I asked about in my question) but rather calculate the differences in proportions 50 times and afterwards test this difference. Here is how I did it:

1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

2. The variance for the difference in proportions in a single completed data set $i$ is calculated as

$$U_i = (p_{i} * (1 - p_{i}))* (\frac{1}{n_1} + \frac{1}{n_2})$$

($p_i$ being the 50 proportions of successes in the 50 data sets, regardless of the group; $n_1$ and $n_2$ are the group sizes)

3. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.008689644$$

4. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001116486$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

5. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.00982846$$

6. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.1310543$$

6. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 3649.726$$

7. Finally, the test-statistic was calculated as

$$\frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - 0.1687879)^2}{0.003769771} = 2.974081$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

The test statistic is referred to an $F$-distribution with $1$ and $v$ degrees of freedom (Schafer, 1997), resulting in a $p$-value of $0.08469283$.

For reference: The $p$-value of my complete case analysis ($n = 98$) was $~ 0.054$.

That's as far as I got. Here's the literature I used:

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: Current practice and guidelines. BMC Medical Research Methodology, 9-57.

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons.

Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data. CRC Press.

I found this source https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-9-57 and would appreciate any comments on the followed procedure:

1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

2. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.093208$$ where $U_i$ are the 50 variances of the 50 single data sets, calculated as

$$U_i = \sqrt{p_{i} * (1 - p_{i}) * (\frac{1}{n_1} + \frac{1}{n_2})}$$

($p_i$ being the 50 proportions of successes in the 50 data sets; $n_1$ and $n_2$ are the group sizes)

3. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001570529$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

4. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.09480994$$

5. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.01718672$$

6. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 171637.2$$

7. Finally, the Wald-test-statistic was calculated as

$$Wald = \frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - diff_{mean})^2}{T} = 0.3004891$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

Is the chosen procedure correct in my case? Or did I miss something? What would be the correct application of Rubin's rule?

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Medical Research Methodology, 9, 57.

I collected some resources and decided to stick with the following procedure - maybe this helps somebody in the future. As I am still not confident with my solution, I would appreciate any comments and/or advise.

Some background: I conducted a study with two groups, one control and one experimental group. My goal was to compare the proportions of successes of both groups, that is, I analysed the difference between the two proportions. In my control group, out of 60 people 21 were successful, 30 were not successful and 9 were missing (ignoring the missing, 41% were successful). In my experimental group, out of 55 people 27 were successful and 8 were missing (ignoring the missing, 57% successful).

At first, I wanted to do 50 z-tests in the 50 completed data sets but I realised, that with multiple imputation I won't perform 50 hypothesis tests (that's what I asked about in my question) but rather calculate the differences in proportions 50 times and afterwards test this difference. Here is how I did it:

1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

2. The variance for a proportion in a single completed data set $i$ is calculated as

$$U_i = \frac{p_{i} * (1 - p_{i})}{n}$$

($p_i$ being the 50 proportions of successes in the 50 data sets, regardless of the group; $n$ is the sample size)

3. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.002167831$$

4. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001570529$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

5. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.09480994$$

6. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.7389596$$

6. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 271.3524$$

7. Finally, the test-statistic was calculated as

$$\frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - 0.1687879)^2}{0.003769771} = 7.557316$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

The test statistic is referred to an $F$-distribution with $1$ and $v$ degrees of freedom (Schafer, 1997), resulting in a $p$-value of $0.00637792$.

For reference: The $p$-value of my complete case analysis ($n = 98$) was $~ 0.054$.

That's as far as I got. Here's the literature I used:

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: Current practice and guidelines. BMC Medical Research Methodology, 9-57.

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons.

Schafer, J. L. (1997). Analysis of Incomplete Multivariate Data. CRC Press.

I found this source https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-9-57 and would appreciate any comments on the followed procedure:

1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

2. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.093208$$ where $U_i$ are the 50 variances of the 50 single data sets, calculated as

$$U_i = \sqrt{p_{i} * (1 - p_{i}) * (\frac{1}{n_1} + \frac{1}{n_2})}$$

($p_i$ being the 50 proportions of successes in the 50 data sets; $n_1$ and $n_2$ are the group sizes)

3. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001570529$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

4. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.09480994$$

5. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.01718672$$

6. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 171637.2$$

7. Finally, the Wald-test-statistic was calculated as

$$Wald = \frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - d_{mean})^2}{T} = 2.768728$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

Is the chosen procedure correct in my case? Or did I miss something? What would be the correct application of Rubin's rule?

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Medical Research Methodology, 9, 57.

I found this source https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-9-57 and would appreciate any comments on the followed procedure:

1. I calculated the mean difference in proportions $diff_{mean}$ over all $m = 50$ imputed data sets, it's approximately $0.169$.

2. I calculated the mean within-variance as $$\overline{U} = \frac{1}{m}\sum _{i=1}^{m} U_i = 0.093208$$ where $U_i$ are the 50 variances of the 50 single data sets, calculated as

$$U_i = \sqrt{p_{i} * (1 - p_{i}) * (\frac{1}{n_1} + \frac{1}{n_2})}$$

($p_i$ being the 50 proportions of successes in the 50 data sets; $n_1$ and $n_2$ are the group sizes)

3. I calculated the between-variance $$B = \frac{1}{m - 1}\sum _{i=1}^{m} (diff_i - diff_{mean})^2 \\ = \frac{1}{50 - 1}\sum _{i=1}^{50} (diff_i - 0.169)^2 \\ = 0.001570529$$ where $diff_i$ are the 50 differences of the proportions of the 50 single data sets.

4. I calculated the total variance as $$T = \overline{U} + (1 + \frac{1}{m}) * B = 0.09480994$$

5. The relative increase in variance was calculated as

$$r = \frac{(1 + \frac{1}{m}) * B}{\overline{U}} = 0.01718672$$

6. The degrees of freedom where calculated as

$$v = (m - 1)(1 + \frac{1}{r})^2 = 171637.2$$

7. Finally, the Wald-test-statistic was calculated as

$$Wald = \frac{(diff_{H_0} - diff_{mean})^2}{T} = \frac{(0 - diff_{mean})^2}{T} = 0.3004891$$

$diff_{H_0} = 0$ as the null hypothesis states there is no difference in proportions.

Is the chosen procedure correct in my case? Or did I miss something? What would be the correct application of Rubin's rule?

Marshall, A., Altman, D. G., Holder, R. L., & Royston, P. (2009). Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Medical Research Methodology, 9, 57.