12
$\begingroup$

I am concerned with the problem that I would like to bootstrap the p-value for an estimate of $\theta$ from multiply imputed (MI) data, but that it is unclear to me how to combine the p-values across MI sets.

For MI data sets, the standard approach to get to the total variance of estimates uses Rubin's rules. See here for a review of pooling MI data sets. The square root of the total variance serves as a standard error estimate of $\theta$. However, for some estimators the total variance has no known closed form or the sampling distribution is not normal. The statistic ${\theta}/{se(\theta)}$ may then not be t-distributed, not even asymptotically.

Therefore, in the complete data case, one alternative option is to bootstrap the statistic to find variance, a p-value and a confidence interval, even if the samling distribution is not normal and its closed form unknown. In the MI case there are then two options:

  • Pool the bootstrapped variance across MI data sets
  • Pool the p-value or confidence bounds across MI data sets

The first option would then again use Rubin's rules. However, I believe that this is problematic, if $\theta$ has a non-normal sampling distribution. In this situation (or more generally, in all situations) the bootstrapped p-value can be used directly. However, in the MI case, this would lead to multiple p-values or confidence intervals, which need to be pooled across MI data sets.

So my question is: how should I pool multiple bootstrapped p-values (or confidence intervals) across multiply imputed data sets?

I would welcome any suggestions on how to proceed, thank you.

Improve this question
$\endgroup$
3
  • $\begingroup$ Perhaps helpful: Missing Data, Imputation and the Bootstrap (Efron 1992) statistics.stanford.edu/sites/default/files/BIO%2520153.pdf $\endgroup$
    – D L Dahly
    Commented Dec 18, 2013 at 12:46
  • $\begingroup$ @DLDahly Hmm, I am not familiar with that paper, but the idea seems to be to bootstrap first, and then perform multiple imputation. The OP appears to be bootstrapping estimates from MI datasets. $\endgroup$
    – tchakravarty
    Commented Dec 18, 2013 at 13:05
  • $\begingroup$ @fgnu Indeed, the standard procedure to get to the total variance of an estimate by bootstrap would be to bootstrap the variance within each MI dataset, and then apply Rubin's rules to pool the bootstrapped variance across MI data sets. $\endgroup$
    – tomka
    Commented Dec 18, 2013 at 13:07

2 Answers 2

Reset to default
8
$\begingroup$

I think both options result in the correct answer. In general, I would prefer method 1 as that preserves the entire distribution.

For method 1, bootstrap the parameter $k$ times within each of the $m$ MI solutions. Then simply mix the $m$ bootstrapped distributions to obtain your final density, now consisting of $k \times m$ samples that include the between-imputation variation. Then treat that as a conventional bootstrap sample to get confidence intervals. Use the Bayesian bootstrap for small samples. I know of no simulation work that investigates this procedure, and this is actually an open problem to be investigated.

For method 2, use the Licht-Rubin procedure. See How to get pooled p-values on tests done in multiple imputed datasets?

Improve this answer
$\endgroup$
3
  • $\begingroup$ +1 - IF the goal is to understand the variability of the estimimates across the MI datasets, I would bootstrap within each MI dataset and look at the total and MI-specific distributions of the parameter. $\endgroup$
    – D L Dahly
    Commented Dec 21, 2013 at 7:54
  • $\begingroup$ @Stef-van-Buuren It seems what D L Dahly suggests is equivalent to pooling the boostrapped variance across MI sets. Would you still prefer your method one (append all bootstrapped data sets) over this 'indirect' approach? $\endgroup$
    – tomka
    Commented Dec 25, 2013 at 14:00
  • $\begingroup$ @tomka. I would certainly do the same as D L Dahly, and study the within and between imputation distributions. In order to integrate both types of distributions, we need to combine them in some way. My suggestion is to simply mix them. $\endgroup$
    – Stef van Buuren
    Commented Dec 26, 2013 at 11:45
7
+25
$\begingroup$

This is not a literature I am familiar with, but one way to approach this might be to ignore the fact that these are bootstrapped p-values, and look at the literature on combining p-values across multiply imputed data sets.

In that case, Li, Meng, Raghunathan, and Rubin (1991) applies. The procedure is based on statistics from each of the imputed datasets, weighted using a measure of the information loss due to imputation. They run into issues related to the joint distribution of the statistics across imputations, and they make some simplifying assumptions.

Of related interest is Meng (1994).

Update

A procedure for combining p-values across multiply imputed datasets is described in the dissertation of Christine Licht, Ch. 4. The idea, which she attributes to Don Rubin, is essentially to transform the p-values to be normally distributed, which can then be combined across MI datasets using the standard rules for combination of z-statistics.

Improve this answer
$\endgroup$
2
  • $\begingroup$ If I understand the Li et al. work correctly, it applies to statistics you get from each MI set. For example, if you get Pearson Chi² on each set, then their rules could be applied to combine it for inference across sets. Also a Wald test could be conducted, for example. But in the case of a bootstrap you do not get a statistic you would pool (but only a p-value). So I am not sure if there is something in Li et al. that could be applied to the bootstrapped p. $\endgroup$
    – tomka
    Commented Dec 19, 2013 at 13:38
  • 1
    $\begingroup$ @tomka I have updated my answer. $\endgroup$
    – tchakravarty
    Commented Dec 20, 2013 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.