Pooling p-values from hypothesis tests after multiple imputation

Question

I'm working on a project that is using some more advanced statistical methods and coding than I'm normally used to and would appreciate some help. The project required me to do multiple imputation, which I did with the mice r package. I now have a mids object containing my 10 imputed datasets (1,200 rows, 123 variables).

I'm working on a table 1 which will have summary statistics and p-values for differences in demographic and other variables across sex (male vs. female). Because my continuous variables are not normally distributed, I was planning on doing Mann Whitney U/Wilcoxon tests to compare them and extract p-values.

My problem lies in how to do this with a multiply imputed dataset and pool the p-values. Is it sufficient to take the median, or should a more advanced method be used? I came across a method here: (How to get pooled p-values on tests done in multiple imputed datasets? -- see comment from Stef van Buuren), but this method states that it is only meant for one-sided tests, and I am performing two-sided tests to detect any difference in characteristics like age, weight, and height across sex.

From what I can tell, a t-test might be robust enough given my sample size (and the method for pooling these p-values appears simpler in R), but all of my continuous variables apart from age are significantly skewed.

Does anyone have an idea of what would be appropriate here?

Answer 1

1. I would compare means, so that you get standard errors and confidence intervals out of the multiple imputations

2. If you really want to do Wilcoxon tests, you can do both the one-sided tests and pool each of them across the imputations to get two pooled one-sided tests. The two-sided p-value is then the sum of the two one-sided p-values

Answer 2

A few thoughts.

First, what you describe as Table 1 often is just supposed to be a summary of your study population. For that Table, it might be most useful to your readers to report the raw male/female values and differences, along with indications of how much data was missing for each variable by sex. You can then (perhaps in supplemental data) describe the similarities and differences between the raw and imputed estimates.

Second, your jump to Wilcoxon tests probably isn't necessary. Think about each male/female comparison as a simple linear model, with a coefficient representing the estimated male-female difference. What's needed for inference is a normal distribution of the sampling distribution of that coefficient estimate, not of the data themselves. You can evaluate how close that sampling distribution comes to a normal distribution by resampling. See this Penn State course page on the Central Limit Theorem, and its examples of resampling. With 1200 observations that assumption should hold pretty well for most or all your variables.

Third, the Wilcoxon test is only a test of stochastic ordering in values between males and females. Although it's sometimes described as a test on medians, that isn't strictly true unless the two distributions have the same shape and only differ by a shift in location. Differences in means, as recommended by Thomas Lumley in another answer, are much more useful.

Fourth, the Wilcoxon test can be considered a special case of ordinal logistic regression. That also doesn't evaluate means or medians; in your situation is would estimate the log-odds of a value from a male being higher than one from a female. The coefficients, estimated by maximum likelihood, have asymptotically normal distributions. Ordinal regression has the further advantage of applying to general multiple regression models.

Fifth, Rubin's Rules work when the estimates being pooled have approximately (multivariate) normal distributions. If (following the second point) the coefficient estimates of male-female differences have close to a normal distribution or if (following the fourth point) you use ordinal regression with asymptotically normal distributions of coefficient estimates, you can use Rubin's Rules directly rather than try to pool p-values

For situations in which the estimates can't be assumed to have an approximately normal distribution, transformations to something approaching a normal distribution can help. See for example Section 5.2 of Stef van Buuren's Flexible Imputation of Missing Data. In particular, the p-value pooling noted in the page you link is based on a way to try to transform the p-value distribution to a normal distribution. I'd expect direct pooling of model coefficients and their variances to be more reliable, particularly with only 10 imputed data sets.