I have multiple set of imputations generated from multiple instances of random forest (such that the predictors are all the variables except the one column to impute). I was referred to Rubin's rule to incorporate all the imputations. However, the guides that I've found online are mainly in the use of R to do so. I want to do it from scratch to understand it better. Also, random forest doesn't assume normality.

My thinking is that let's say I have 5 of imputation sets. Why don't I just take the mean for each of the imputations?


1 Answer 1


After multiple imputation of data sets (MI) and analyzing each of the imputed sets separately, Rubin's rules do have you take the mean over those imputations as the point estimate. For inference, confidence intervals and so forth, you then determine the overall variance of the point estimate as a combination of within-imputation and between-imputation variances. This paper by Marshall, Altman, et al. summarizes the rules nicely:

"For a single population parameter of interest, $Q$, e.g. a regression coefficient, the MI overall point estimate is the average of the $m$ estimates of $Q$ from the imputed datasets, $\bar Q = \frac{1}{m} \sum_{i=1}^m \hat Q_i$. The associated total variance for this overall MI estimate is $T = \bar U + \left( 1+\frac{1}{m} \right) B$, where $\bar U = \frac{1}{m} \sum_{i=1}^m U_i$ is the estimated within imputation variance and $B=\frac{1}{m-1} \sum_{i=1}^m \left( \hat Q_i - \bar Q\right)^2$ is the between imputation variance. Inflating the between imputation variance by a factor $1/m$ reflects the extra variability as a consequence of imputing the missing data using a finite number of imputations instead of an infinite number of imputations. When $B$ dominates $\bar U$ greater efficiency, and hence more accurate estimates, can be obtained by increasing $m$. Conversely, when $\bar U$ dominates $B$, little is gained from increasing $m$."

The paper then goes on to show how to apply these rules to multiple coefficients, situations in which coefficient estimates might not have normal distributions, hypothesis testing, and so on.

  • $\begingroup$ What is the general rule on choosing the imputation set? Should we choose the set which yield the lowest variance ? $\endgroup$ Jul 13, 2020 at 0:55
  • $\begingroup$ @daddymaterial you don't choose a set; you analyze all imputed sets in parallel and combine the estimates from them. Use a method that incorporates the uncertainty in the parameters used for the imputation: "Bayesian methods draw the parameters directly from their posterior distributions ... bootstrap methods resample the observed data and re-estimate the parameters from the resampled data." 5 imputed sets are good to start, increasing if needed; see the van Buuren FIMD site. $\endgroup$
    – EdM
    Jul 13, 2020 at 12:57
  • $\begingroup$ Are the variances really correct? I have implemented this in R. However, I bootstrapped the estimates as a cross-check and got standard errors half as large. $\endgroup$
    – jay.sf
    May 20 at 7:36
  • $\begingroup$ @jay.sf I'm pretty sure that these formulas agree with what's in Section 2.3.2 of Flexible imputation of missing data (FIMD), written by the author of the mice package you used in the linked implementation. Combining bootstrapping with multiple imputation can be tricky; see section 5.4.3 of FIMD for one example. Post a new question with a reproducible example if you'd like to clarify what's going on with your bootstrapping. $\endgroup$
    – EdM
    May 20 at 13:23
  • $\begingroup$ @EdM I've posted a question, thanks. $\endgroup$
    – jay.sf
    2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.