Imputation with Random Forests

I have two questions on using random forest (specifically randomForest in R) for missing value imputation (in the predictor space).

1) How does the imputation algorithm work - specifically how and why is the class label required for imputation? is the proximity matrix which serves to weight the average value to impute a missing value defined separately by class?

2) If the class label is needed to impute missing values - how can this be used to impute missing values for new data that you are trying to predict?

The basic idea is to do a quick replacement of missing data and then iteratively improve the missing imputation using proximity. To work with unlabeled data, just replicate the data with all labels, and then treat it as labeled data.

The fraction of trees for which a pair of observations share a terminal node gives the proximity matrix, and so explicitly uses the class label.

Training set:

1. Replace missing values by the average value.
2. Repeat until satisfied:

a. Using imputed values calculated so far, train a random forest.

b. Compute the proximity matrix.

c. Using the proximity as the weight, impute missing values as the weighted average of non-missing values.

Test set:

1. If labels exist, use the imputation derived from test data.
2. If data is unlabeled, replicate the test set with a copy for each class label and proceed as with labeled data.

Here, (weighted) average refers to (weighted) median for numerical variables and (weighted) mode for categorical variables. 4-6 iterations are recommended in the references.

• I'd be interested to know more about whether this algorithm can be adapted for multiple imputation, and whether it would have the right amount of variability and account for imputation model uncertainty. Feb 6 '13 at 4:11
• Frank, from the description of it, I doubt it would have enough variability. Drawing hot deck from a terminal class may do the trick. If the tree growing algorithm tends to overfit, the variability would still be suppressed, but not by as much as when you are using a conditional mean or a conditional quantile. Again, that's my gut feeling concerning how imputation methods work, in general. Feb 6 '13 at 13:24
• Cohoz, thank you this confirms what I had since learned. The issue is the random forest being built using the target variable. There is a missForest package in R with paper that can be used for unsupervised imputation: ncbi.nlm.nih.gov/pubmed/22039212 Feb 6 '13 at 23:16
• If i were to ask, does sklearn.ensemble.RandomForestClassifier does this process on training data or will it just ignore it and I have to do it myself? Apr 2 '20 at 15:39

I have tried using Random Forest for multiple imputation in MICE to handle missing data in survival analysis. I used bootstrapping to account for sampling variability in the imputation models. I found that Random Forest MICE performed better than parametric MICE when there were interactions between predictor variables that were not included in the imputation model.

The CALIBERrfimpute package provides a function for Random Forest imputation in MICE:
http://cran.r-project.org/web/packages/CALIBERrfimpute/index.html

This is an article describing tests of the method on simulated data and a real epidemiological dataset:
http://dx.doi.org/10.1093/aje/kwt312

• Welcome to the site, @user37364. Thanks for these links. Would you mind elaborating on them a little, in case of linkrot & so future readers can judge before clicking if they want to pursue them? Jan 15 '14 at 21:05
• Hi @user37364! I've seen the paper you present, however, I'm not able to apply random forest with mice in my dataset. I posted a question here: stackoverflow.com/questions/24239595/…. If you have experience with MICE, do you have any idea how to solve these errors? Thanks Jun 18 '14 at 13:07
• Hello, I just came across your paper, and then this thread. How has the method held up over the past year and a half? Any wrinkles discovered? May 17 '16 at 22:10