10
$\begingroup$

I have two questions on using random forest (specifically randomForest in R) for missing value imputation (in the predictor space).

1) How does the imputation algorithm work - specifically how and why is the class label required for imputation? is the proximity matrix which serves to weight the average value to impute a missing value defined separately by class?

2) If the class label is needed to impute missing values - how can this be used to impute missing values for new data that you are trying to predict?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
10
$\begingroup$

The basic idea is to do a quick replacement of missing data and then iteratively improve the missing imputation using proximity. To work with unlabeled data, just replicate the data with all labels, and then treat it as labeled data.

The fraction of trees for which a pair of observations share a terminal node gives the proximity matrix, and so explicitly uses the class label.

Training set:

  1. Replace missing values by the average value.

  2. Repeat until satisfied:

    a. Using imputed values calculated so far, train a random forest.

    b. Compute the proximity matrix.

    c. Using the proximity as the weight, impute missing values as the weighted average of non-missing values.

Test set:

  1. If labels exist, use the imputation derived from test data.
  2. If data is unlabeled, replicate the test set with a copy for each class label and proceed as with labeled data.

Here, (weighted) average refers to (weighted) median for numerical variables and (weighted) mode for categorical variables. 4-6 iterations are recommended in the references.

R documentation (pdf), Breiman's manual v4.0 (pdf), Breiman's RF page

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ I'd be interested to know more about whether this algorithm can be adapted for multiple imputation, and whether it would have the right amount of variability and account for imputation model uncertainty. $\endgroup$
    – Frank Harrell
    Feb 6, 2013 at 4:11
  • 1
    $\begingroup$ Frank, from the description of it, I doubt it would have enough variability. Drawing hot deck from a terminal class may do the trick. If the tree growing algorithm tends to overfit, the variability would still be suppressed, but not by as much as when you are using a conditional mean or a conditional quantile. Again, that's my gut feeling concerning how imputation methods work, in general. $\endgroup$
    – StasK
    Feb 6, 2013 at 13:24
  • 1
    $\begingroup$ Cohoz, thank you this confirms what I had since learned. The issue is the random forest being built using the target variable. There is a missForest package in R with paper that can be used for unsupervised imputation: ncbi.nlm.nih.gov/pubmed/22039212 $\endgroup$
    – B_Miner
    Feb 6, 2013 at 23:16
  • 1
    $\begingroup$ If i were to ask, does sklearn.ensemble.RandomForestClassifier does this process on training data or will it just ignore it and I have to do it myself? $\endgroup$
    – Abhishta Gatya
    Apr 2, 2020 at 15:39
  • $\begingroup$ stefvanbuuren.name/fimd/sec-cart.html "The missForest method (Stekhoven and Bühlmann 2011) [...] is popular, presumably because it produces a single complete dataset, which at the same time is the reason why it fails as a scientific method. The missForest method does not account for the uncertainty caused by the missing data, treats the imputed data as if they were real (which they are not), and thus invents information." $\endgroup$
    – user16263
    Apr 28 at 17:38
4
$\begingroup$

I have tried using Random Forest for multiple imputation in MICE to handle missing data in survival analysis. I used bootstrapping to account for sampling variability in the imputation models. I found that Random Forest MICE performed better than parametric MICE when there were interactions between predictor variables that were not included in the imputation model.

The CALIBERrfimpute package provides a function for Random Forest imputation in MICE:
http://cran.r-project.org/web/packages/CALIBERrfimpute/index.html

This is an article describing tests of the method on simulated data and a real epidemiological dataset:
http://dx.doi.org/10.1093/aje/kwt312

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ Welcome to the site, @user37364. Thanks for these links. Would you mind elaborating on them a little, in case of linkrot & so future readers can judge before clicking if they want to pursue them? $\endgroup$
    – gung - Reinstate Monica
    Jan 15, 2014 at 21:05
  • $\begingroup$ Hi @user37364! I've seen the paper you present, however, I'm not able to apply random forest with mice in my dataset. I posted a question here: stackoverflow.com/questions/24239595/…. If you have experience with MICE, do you have any idea how to solve these errors? Thanks $\endgroup$
    – psoares
    Jun 18, 2014 at 13:07
  • $\begingroup$ Hello, I just came across your paper, and then this thread. How has the method held up over the past year and a half? Any wrinkles discovered? $\endgroup$
    – generic_user
    May 17, 2016 at 22:10
0
$\begingroup$

The algorithm implemented in mice.impute.rf is described in Appendix A.1 of Doove, L.L., van Buuren, S., Dusseldorp, E. (2014), Recursive partitioning for missing data imputation in the presence of interaction Effects. Computational Statistics & Data Analysis, 72, 92-104. Available here: https://stefvanbuuren.name/publications/2014%20Recursive%20partitioning%20-%20CSDA.pdf

In short, for each tree in the ensemble, the algorithm determines the leave from which a missing observation would be predicted, then the elements of all these resulting leaves are pooled together and the imputed value is randomly selected from them (the pooled elements make up a predictive distribution). There is no averaging across trees in order to preserve uncertainty of the imputed value.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.