Normally, I used a test set to calculate the RMSE of my RandomForest model. But currently I am using the whole data set in the Random Forest. I want to validate (RMSE) my model with the "out of bag error" (so an out of bag error, calculated as RMSE). Is that possible?

I am looking specific for the RMSE, since I evaluate my other models with this metric.

If I run (R, package: RandomForest):

Rf_model <- randomForest(target ~., data = whole_data) 

randomForest(formula = target ~ ., data = whole_data) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 27

          Mean of squared residuals: 0.05206834
                    % Var explained: 94.61

it returns the Mean of squared residuals. But how can I get the RMSE?

  • $\begingroup$ It should be possible, I know that with the train() function from the caret package this is possible. $\endgroup$ Sep 28, 2018 at 10:42
  • $\begingroup$ Try using library(ranger) and this result: doi.org/10.1016/j.patrec.2022.04.031 $\endgroup$
    – Zen
    May 13, 2022 at 1:05

2 Answers 2


I think I got the solution for the OOB RMSE, using keep.inbag=T from randomForest.

First you can use predict in order to get the predictions from the model for your response, than simply evaluate using the RMSE formula:

Rf_model <- randomForest(mpg ~., data = mtcars) 

rf_pred <- predict(Rf_model, mtcars) # predictions

sqrt(sum(rf_pred - mtcars$mpg)^2) #RMSE
#[1] 0.1781314

You can get fancy and make a custom rmse function to call:

rmse_function <- function(pred, actual) {
  sqrt(sum(pred - actual)^2)

rmse_function(rf_pred, mtcars$mpg)
#[1] 0.1781314

But this is the overall RMSE on train data. Not the OOB.

We can probably calculate the OOB RMSE by keeping track of which observation is kept "outside" in each n_tree in the forest.

Then we can use this to subset the data in order to make the prediction using only these rows. (The out of bag obs)

Following this idea, we will have to make n_tree predictions, using only the subset of observations that for each tree is kept "out".

We will have then n_tree RMSE, and we can average those to have an averate RMSE of the OOB observations.

n_tree = 50
Rf_model <- randomForest(mpg ~., ntree = n_tree, data = mtcars, keep.inbag=T)  # we use keep.inbag = T

inbag <- lapply(1:n_tree, function(x) which(Rf_model[["inbag"]][ ,x] == 0)) # we get only the "zeros"
# to look inside use View(Rf_model[["inbag"]]), I think that the zeros are the OOB

rf_pred <- lapply(inbag, function(x) predict(Rf_model, mtcars[x, ])) # predictions

(oob_err <- map2_dbl(rf_pred, inbag, function(x, y) rmse_function(x, mtcars[y, ]$mpg)))
# [1] 1.03926667 0.01556667 2.98096667 1.27210000 1.86380000 2.25883333 3.49130000 0.18763333 1.59326667 0.11236667
# [11] 6.92163333 0.40183333 3.36586667 1.19960000 1.31833333 2.88373333 4.48326667 1.67406667 6.92566667 8.51793333
# [21] 3.32893333 0.65510000 3.87440000 1.89276667 3.51290000 3.13026667 4.81453333 0.59756667 1.56783333 6.12180000
# [31] 3.54490000 0.57406667 0.20236667 2.20220000 0.23226667 1.61360000 0.32690000 1.86300000 3.38393333 3.33723333
# [41] 1.43760000 6.63860000 0.13120000 1.48580000 1.32950000 2.85310000 2.01306667 2.16363333 4.80706667 1.74310000

mean(oob_err) # mean of the RMSEs
#[1] 2.477725
  • $\begingroup$ I think there could be some issues here. You are getting predictions from the average of all of your trees with the statement predict(Rf_model, mtcars[x, ]). I think instead you should be using the predict.all = TRUE argument there to get the individual tree predictions, and then you can extract the particular tree that corresponds to the OOB observations. I think the other issue is that the RMSE calculation should probably be based on first getting predictions for each observation (averaging trees where it is OOB), and only then doing the RMSE calculation. $\endgroup$
    – bzki
    Dec 6, 2019 at 18:52

The Mean of squared residuals: 0.05206834 in your output is the out-of-bag MSE estimate. Just take the square root:

sqrt (tail (Rf_model$mse, 1))

(Apparently, $mse stores the oob MSE observed for bagging 1 : n trees, the last one is the one we need.)

You can double check by manually calculating RMSE from the oob predictions:

sqrt (mean ((Rf_model$predicted - whole_data$target)^2) 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.