1
$\begingroup$

I want to use the R missForest() function at work to perform missing value imputation. However, after reading up on the algorithm more, I can't decide how to impute future/test data - running the algorithm on just the test data, or run it on train + test combined.

I have heard many times that you training & test datasets should not inform one another at all, but this algorithm is an example of where I think that doing so would increase predictive performance a lot, and crucially I see no reason why I wouldn't be able to apply this same approach in production for future unseen data.

Here's an example of what i'm proposing. First I do a train/test split and introduce missing values:

library(datasets)
library(missForest)
library(randomForest)
data("iris")

# train/test:
set.seed(2)
train_index <- sample(nrow(iris), 120)
feature_train <- as.matrix(iris[train_index, 1:4])
feature_test <- as.matrix(iris[-train_index, 1:4])

# introduce NA's to both:
feature_train[sample(length(feature_train), 40)] <- NA
feature_test[sample(length(feature_test), 10)] <- NA

I impute the training data and build a model on the imputed data:

imp_feature_train <- missForest(feature_train)$ximp

rf <- randomForest(x = imp_feature_train, y = iris$Species[train_index])

Method 1:

For the test data, this is the method that I prefer. It uses the imputed training data to inform missForest() in imputing the test data. Is there any issue with this, as long as (if the model was put into production) I also used this training data to inform any future data?

feature_train_test <- rbind(feature_test, imp_feature_train)

imp_feature_test_v1 <- missForest(feature_train_test)$ximp[1:nrow(feature_test), ]

pred_test_v1 <- predict(rf, imp_feature_test_v1)
sum(pred_test_v1 == iris$Species[-train_index]) # 27

The reason I want to use this approach: imagine your training dataset is 1M observations, and you are building a model that must be ran hourly on 1k observations. The 'quality' of the missing value imputations will surely be higher if you use those 1M observations to 'inform' the missForest() algorithm, as apposed to simply running the algorithm on the 1k observations alone.


Method 2:

This imputes the train and test datasets completely separately. I imagine that performance would degrade considerably as the test dataset gets smaller:

imp_feature_test_v2 <- missForest(feature_test)$ximp

pred_test_v2 <- predict(rf, imp_feature_test_v2)
sum(pred_test_v2 == iris$Species[-train_index]) # 26

Thanks in advance.

$\endgroup$
0

1 Answer 1

0
$\begingroup$

In Method 1, I think it is a kind of target leakage because you use test data for predicting each NA value.

How about using this modified dataset rbind(feature_test[each_row], imp_feature_train)? just predict each 1 data sample. I am searching for about good imputation example by using missforest.

$\endgroup$
3
  • $\begingroup$ Hey, I actually ended up doing a fair bit of work on missforest after asking this question. You might find this useful: kaggle.com/lmorgan95/missforest-the-best-imputation-algorithm $\endgroup$ Commented Jun 1, 2021 at 17:07
  • $\begingroup$ Thanks for the article, @LiamMorgan! Probably naive question, but is it safe to say that your missforest implementation can efficiently impute/recover missing data? $\endgroup$
    – surlac
    Commented Jul 4, 2021 at 6:44
  • 1
    $\begingroup$ Random forests are notorious for overfitting so I would recommend caution in using it for imputation. Be sure to verify using simulation the accuracy of confidence interval coverage when using this approach. $\endgroup$ Commented Nov 25, 2023 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.