Random Forest "out of bag" RMSE

Question

Normally, I used a test set to calculate the RMSE of my RandomForest model. But currently I am using the whole data set in the Random Forest. I want to validate (RMSE) my model with the "out of bag error" (so an out of bag error, calculated as RMSE). Is that possible?

I am looking specific for the RMSE, since I evaluate my other models with this metric.

If I run (R, package: RandomForest):

Rf_model <- randomForest(target ~., data = whole_data) 

Rf_model
Call:
randomForest(formula = target ~ ., data = whole_data) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 27

          Mean of squared residuals: 0.05206834
                    % Var explained: 94.61

it returns the Mean of squared residuals. But how can I get the RMSE?

Answer 1

I think I got the solution for the OOB RMSE, using keep.inbag=T from randomForest.

First you can use predict in order to get the predictions from the model for your response, than simply evaluate using the RMSE formula:

Rf_model <- randomForest(mpg ~., data = mtcars) 

rf_pred <- predict(Rf_model, mtcars) # predictions

sqrt(sum(rf_pred - mtcars$mpg)^2) #RMSE
#[1] 0.1781314

You can get fancy and make a custom rmse function to call:

rmse_function <- function(pred, actual) {
  sqrt(sum(pred - actual)^2)
}

rmse_function(rf_pred, mtcars$mpg)
#[1] 0.1781314

But this is the overall RMSE on train data. Not the OOB.

We can probably calculate the OOB RMSE by keeping track of which observation is kept "outside" in each n_tree in the forest.

Then we can use this to subset the data in order to make the prediction using only these rows. (The out of bag obs)

Following this idea, we will have to make n_tree predictions, using only the subset of observations that for each tree is kept "out".

We will have then n_tree RMSE, and we can average those to have an averate RMSE of the OOB observations.

n_tree = 50
Rf_model <- randomForest(mpg ~., ntree = n_tree, data = mtcars, keep.inbag=T)  # we use keep.inbag = T

inbag <- lapply(1:n_tree, function(x) which(Rf_model[["inbag"]][ ,x] == 0)) # we get only the "zeros"
# to look inside use View(Rf_model[["inbag"]]), I think that the zeros are the OOB


rf_pred <- lapply(inbag, function(x) predict(Rf_model, mtcars[x, ])) # predictions


(oob_err <- map2_dbl(rf_pred, inbag, function(x, y) rmse_function(x, mtcars[y, ]$mpg)))
# [1] 1.03926667 0.01556667 2.98096667 1.27210000 1.86380000 2.25883333 3.49130000 0.18763333 1.59326667 0.11236667
# [11] 6.92163333 0.40183333 3.36586667 1.19960000 1.31833333 2.88373333 4.48326667 1.67406667 6.92566667 8.51793333
# [21] 3.32893333 0.65510000 3.87440000 1.89276667 3.51290000 3.13026667 4.81453333 0.59756667 1.56783333 6.12180000
# [31] 3.54490000 0.57406667 0.20236667 2.20220000 0.23226667 1.61360000 0.32690000 1.86300000 3.38393333 3.33723333
# [41] 1.43760000 6.63860000 0.13120000 1.48580000 1.32950000 2.85310000 2.01306667 2.16363333 4.80706667 1.74310000

mean(oob_err) # mean of the RMSEs
#[1] 2.477725

Answer 2

The Mean of squared residuals: 0.05206834 in your output is the out-of-bag MSE estimate. Just take the square root:

sqrt (tail (Rf_model$mse, 1))

(Apparently, $mse stores the oob MSE observed for bagging 1 : n trees, the last one is the one we need.)

You can double check by manually calculating RMSE from the oob predictions:

sqrt (mean ((Rf_model$predicted - whole_data$target)^2)