perform Random Forest AFTER multiple imputation with MICE

Question

i wanted to build a prediction model. Since my data had some missing data, I imputed data with the MICE algorithm. After that I wanted to do a regression with Random Forest.

Now I'm kinda stuck because:

I wanted to do Multiple Imputation with MICE because I wanted to show consideration for the variance of the missing variables in my model. So I imputed 5 data sets with MICE.

If i wanted to do a glm, I would build 5 models(for each imputed data set) and then pool them together. (Meaning in the end i have 1 model and the variance of my parameters will be higher)

Now what I want to do is to build a random forest. But I just can't find any strategies for this. Since RF doesn't have parameter estimations, I can't pool them together...

Has anyone worked on this before? or any advices what I should do?

Best wishes and thank you in advance! I really appreciate any help and answers

Ching

Answer 1

This is not a direct answer to your question, and I don't have enough reputation to comment, but one thing you can do is use the Machine Learning in R package. There are many random forest learner implementations there that can use data with missing values. You can also tune the learners based on what your dataset is.

Links to the package and documentation are on the main tutorial page, here:

https://mlr-org.github.io/mlr-tutorial/release/html/index.html

Also, consider that answering your question becomes much easier if you provide a sample of your dataset.

If you need a direct answer, looping a series of RF calls on the imputed datasets might work. E.g. if you have five imputations:

res = data.frame(matrix(0,nrow=nrow(test),ncol=5)
for (i in 1:5){
  data = complete(miceResult, 1)
  rf.res = cforest(data,formula ~ [which formula?])
  res[,i] = predict(rf.res, test)
}

Then you can pool the results by majority voting or averaging, depending on your dataset. You can also group the 5 imputations together and train the learner with the combined dataset. Both methods are suboptimal, however.

Hope this helps.

Answer 2

The combine function in randomForest makes it possible to combine multiple randomForest objects.

Prepare data:

set.seed(1234)

X1 <- rnorm(100, 120, 16)
X2 <- X1 + rnorm(100, 200, 10)
X3 <- 0.8*X2 + rnorm(100, 140, 12)
Y <- factor(as.numeric(X1 > 125))

dat.test <- data.frame(Y, X1, X2, X3)

# Impose missingness
Y[runif(100) < 0.5] <- NA
X1[runif(100) < 0.5] <- NA
X2[runif(100) < 0.5] <- NA
X3[runif(100) < 0.5] <- NA

dat <- data.frame(Y, X1, X2, X3)

Impute missing data:

library(mice)
mice <- mice(dat, m = 10, method = "rf")

impdat <- NULL # allocate empty list of imputations

for (m in 1:10){impdat[[m]] <- complete(mice, m)} # export imputations

Now train m models on m complete data sets:

library(randomForest)
rf <- NULL
for (m in 1:10){rf[[m]] <- randomForest(Y ~ ., data = impdat[[m]])}

Option 1

The combine function in randomForest can aggregate same-size trees:

body(combine)[[4]] <- substitute(rflist <- (...))
rf.all <- combine(rf)

Where rf.all is your 'pooled' model. If we test it:

predictions <- predict(rf.all, within(dat.test, rm("Y")))
table(dat.test$Y, predictions)

    0  1
  0 70  1
  1  2 27

We find the predictions are quite accurate.

Option 2

A second option is to pool together the votes from each model:

votes <- list()
for (m in 1:10){votes[[m]] <- predict(rf[[m]], 
                             within(dat.test, rm("Y")), 
                             type = "vote")}
votes <- Reduce('+', votes)
predictions <- NULL
for (i in 1:nrow(votes))
{if (votes[i,1] < votes[i,2])
    {predictions[i] = 1} else {predictions[i] = 0}}

Note the choice of type = "vote" in the argument of predict. Other functions might require type = "prob".

> table(dat.test$Y, predictions) # mostly accurate
   predictions
     0  1
  0 70  1
  1  2 27

The confusion matrix is the same.

Pooling votes is a general approach for tree-based models that should satisfy for ranger objects and gbm objects.

If the goal is regression rather than classification, pooling the votes is very similar, but set type = "response" in predict (the default).

rf <- NULL
for (m in 1:10){rf[[m]] <- randomForest(X1 ~ ., data = impdat[[m]])}
predictions <- list()
for (m in 1:10){predictions[[m]] <- predict(rf[[m]], 
                                      within(dat.test, rm("X1")),
                                      type = "response")}

predictions <- Reduce('+', predictions)/10 # divide by m

Calculate mean square error:

> mean((predictions - dat.test$X1)^2)
[1] 64.78884