# Pooling p-values from hypothesis tests after multiple imputation

I'm working on a project that is using some more advanced statistical methods and coding than I'm normally used to and would appreciate some help. The project required me to do multiple imputation, which I did with the mice r package. I now have a mids object containing my 10 imputed datasets (1,200 rows, 123 variables).

I'm working on a table 1 which will have summary statistics and p-values for differences in demographic and other variables across sex (male vs. female). Because my continuous variables are not normally distributed, I was planning on doing Mann Whitney U/Wilcoxon tests to compare them and extract p-values.

My problem lies in how to do this with a multiply imputed dataset and pool the p-values. Is it sufficient to take the median, or should a more advanced method be used? I came across a method here: (How to get pooled p-values on tests done in multiple imputed datasets? -- see comment from Stef van Buuren), but this method states that it is only meant for one-sided tests, and I am performing two-sided tests to detect any difference in characteristics like age, weight, and height across sex.

From what I can tell, a t-test might be robust enough given my sample size (and the method for pooling these p-values appears simpler in R), but all of my continuous variables apart from age are significantly skewed.

Does anyone have an idea of what would be appropriate here?

• The mice package comes with a lot of vignettes (see search.r-project.org/CRAN/refmans/mice/html/mice.html ). Do none of those help? Also, see the with.mids help page. It doesn't have an example of Wilcoxon, but it does seem to imply that you can just put it into the with.mids function. The thread that you cite is about Amelia, which is an older function and may not have the same abilities as mice and with.mids. Commented Dec 14, 2023 at 20:45

1. I would compare means, so that you get standard errors and confidence intervals out of the multiple imputations

2. If you really want to do Wilcoxon tests, you can do both the one-sided tests and pool each of them across the imputations to get two pooled one-sided tests. The two-sided p-value is then the sum of the two one-sided p-values

• Thank you! So you're saying a t-test should be robust enough to handle skewedness in my data? It's much easier to pool t-tests after mice, so that would be very helpful. Commented Dec 21, 2023 at 18:13

A few thoughts.

First, what you describe as Table 1 often is just supposed to be a summary of your study population. For that Table, it might be most useful to your readers to report the raw male/female values and differences, along with indications of how much data was missing for each variable by sex. You can then (perhaps in supplemental data) describe the similarities and differences between the raw and imputed estimates.

Second, your jump to Wilcoxon tests probably isn't necessary. Think about each male/female comparison as a simple linear model, with a coefficient representing the estimated male-female difference. What's needed for inference is a normal distribution of the sampling distribution of that coefficient estimate, not of the data themselves. You can evaluate how close that sampling distribution comes to a normal distribution by resampling. See this Penn State course page on the Central Limit Theorem, and its examples of resampling. With 1200 observations that assumption should hold pretty well for most or all your variables.

Third, the Wilcoxon test is only a test of stochastic ordering in values between males and females. Although it's sometimes described as a test on medians, that isn't strictly true unless the two distributions have the same shape and only differ by a shift in location. Differences in means, as recommended by Thomas Lumley in another answer, are much more useful.

Fourth, the Wilcoxon test can be considered a special case of ordinal logistic regression. That also doesn't evaluate means or medians; in your situation is would estimate the log-odds of a value from a male being higher than one from a female. The coefficients, estimated by maximum likelihood, have asymptotically normal distributions. Ordinal regression has the further advantage of applying to general multiple regression models.

Fifth, Rubin's Rules work when the estimates being pooled have approximately (multivariate) normal distributions. If (following the second point) the coefficient estimates of male-female differences have close to a normal distribution or if (following the fourth point) you use ordinal regression with asymptotically normal distributions of coefficient estimates, you can use Rubin's Rules directly rather than try to pool p-values

For situations in which the estimates can't be assumed to have an approximately normal distribution, transformations to something approaching a normal distribution can help. See for example Section 5.2 of Stef van Buuren's Flexible Imputation of Missing Data. In particular, the p-value pooling noted in the page you link is based on a way to try to transform the p-value distribution to a normal distribution. I'd expect direct pooling of model coefficients and their variances to be more reliable, particularly with only 10 imputed data sets.

• I see! Thank you very much for the detailed response. I think in my field the norm is still to use t-tests only when the variable(s) themselves are normally distributed, and transform them or do non-parametric tests otherwise, so I've been stuck in that thinking as well. Commented Dec 21, 2023 at 18:14
• @smirza this page covers the issues in choosing between t-test-type tests and nonparametric tests in some detail.
– EdM
Commented Dec 21, 2023 at 19:01