You can use predict in order to get the predictions from the model for your response, than simply evaluate using the RMSE formula:

Rf_model <- randomForest(mpg ~., data = mtcars) 

rf_pred <- predict(Rf_model, mtcars) # predictions

sqrt(sum(rf_pred - mtcars$mpg)^2) #RMSE
#[1] 0.1781314

You can get fancy and make a custom rmse function to call:

rmse_function <- function(pred, actual) {
  sqrt(sum(pred - actual)^2)
}

rmse_function(rf_pred, mtcars$mpg)
#[1] 0.1781314

I think I got the solution for the OOB RMSE, using keep.inbag=T from randomForest.

First you can use predict in order to get the predictions from the model for your response, than simply evaluate using the RMSE formula:

Rf_model <- randomForest(mpg ~., data = mtcars) 

rf_pred <- predict(Rf_model, mtcars) # predictions

sqrt(sum(rf_pred - mtcars$mpg)^2) #RMSE
#[1] 0.1781314

You can get fancy and make a custom rmse function to call:

rmse_function <- function(pred, actual) {
  sqrt(sum(pred - actual)^2)
}

rmse_function(rf_pred, mtcars$mpg)
#[1] 0.1781314

But this is the overall RMSE on train data. Not the OOB.

We can probably calculate the OOB RMSE by keeping track of which observation is kept "outside" in each n_tree in the forest.

Then we can use this to subset the data in order to make the prediction using only these rows. (The out of bag obs)

Following this idea, we will have to make n_tree predictions, using only the subset of observations that for each tree is kept "out".

We will have then n_tree RMSE, and we can average those to have an averate RMSE of the OOB observations.

n_tree = 50
Rf_model <- randomForest(mpg ~., ntree = n_tree, data = mtcars, keep.inbag=T)  # we use keep.inbag = T

inbag <- lapply(1:n_tree, function(x) which(Rf_model[["inbag"]][ ,x] == 0)) # we get only the "zeros"
# to look inside use View(Rf_model[["inbag"]]), I think that the zeros are the OOB


rf_pred <- lapply(inbag, function(x) predict(Rf_model, mtcars[x, ])) # predictions


(oob_err <- map2_dbl(rf_pred, inbag, function(x, y) rmse_function(x, mtcars[y, ]$mpg)))
# [1] 1.03926667 0.01556667 2.98096667 1.27210000 1.86380000 2.25883333 3.49130000 0.18763333 1.59326667 0.11236667
# [11] 6.92163333 0.40183333 3.36586667 1.19960000 1.31833333 2.88373333 4.48326667 1.67406667 6.92566667 8.51793333
# [21] 3.32893333 0.65510000 3.87440000 1.89276667 3.51290000 3.13026667 4.81453333 0.59756667 1.56783333 6.12180000
# [31] 3.54490000 0.57406667 0.20236667 2.20220000 0.23226667 1.61360000 0.32690000 1.86300000 3.38393333 3.33723333
# [41] 1.43760000 6.63860000 0.13120000 1.48580000 1.32950000 2.85310000 2.01306667 2.16363333 4.80706667 1.74310000

mean(oob_err) # mean of the RMSEs
#[1] 2.477725