0
$\begingroup$

I want to use the R missForest() function at work to perform missing value imputation. However, after reading up on the algorithm more, I can't decide how to impute future/test data - running the algorithm on just the test data, or run it on train + test combined.

I have heard many times that you training & test datasets should not inform one another at all, but this algorithm is an example of where I think that doing so would increase predictive performance a lot, and crucially I see no reason why I wouldn't be able to apply this same approach in production for future unseen data.

Here's an example of what i'm proposing. First I do a train/test split and introduce missing values:

library(datasets)
library(missForest)
library(randomForest)
data("iris")

# train/test:
set.seed(2)
train_index <- sample(nrow(iris), 120)
feature_train <- as.matrix(iris[train_index, 1:4])
feature_test <- as.matrix(iris[-train_index, 1:4])

# introduce NA's to both:
feature_train[sample(length(feature_train), 40)] <- NA
feature_test[sample(length(feature_test), 10)] <- NA

I impute the training data and build a model on the imputed data:

imp_feature_train <- missForest(feature_train)$ximp

rf <- randomForest(x = imp_feature_train, y = iris$Species[train_index])

Method 1:

For the test data, this is the method that I prefer. It uses the imputed training data to inform missForest() in imputing the test data. Is there any issue with this, as long as (if the model was put into production) I also used this training data to inform any future data?

feature_train_test <- rbind(feature_test, imp_feature_train)

imp_feature_test_v1 <- missForest(feature_train_test)$ximp[1:nrow(feature_test), ]

pred_test_v1 <- predict(rf, imp_feature_test_v1)
sum(pred_test_v1 == iris$Species[-train_index]) # 27

The reason I want to use this approach: imagine your training dataset is 1M observations, and you are building a model that must be ran hourly on 1k observations. The 'quality' of the missing value imputations will surely be higher if you use those 1M observations to 'inform' the missForest() algorithm, as apposed to simply running the algorithm on the 1k observations alone.


Method 2:

This imputes the train and test datasets completely separately. I imagine that performance would degrade considerably as the test dataset gets smaller:

imp_feature_test_v2 <- missForest(feature_test)$ximp

pred_test_v2 <- predict(rf, imp_feature_test_v2)
sum(pred_test_v2 == iris$Species[-train_index]) # 26

Thanks in advance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.