I think I got the solution for the OOB RMSE, using `keep.inbag=T` from `randomForest`.

First you can use `predict` in order to get the predictions from the model for your response, than simply evaluate using the [RMSE][1] formula:

    Rf_model <- randomForest(mpg ~., data = mtcars) 
    
    rf_pred <- predict(Rf_model, mtcars) # predictions
    
    sqrt(sum(rf_pred - mtcars$mpg)^2) #RMSE
    #[1] 0.1781314


You can get fancy and make a custom rmse function to call:

    rmse_function <- function(pred, actual) {
      sqrt(sum(pred - actual)^2)
    }
    
    rmse_function(rf_pred, mtcars$mpg)
    #[1] 0.1781314

But this is the overall RMSE on train data. **Not the OOB**.

We can probably calculate the OOB RMSE by keeping track of which observation is kept "outside" in each `n_tree` in the forest. 

Then we can use this to subset the data in order to make the prediction using *only* these rows. (The out of bag obs)

Following this idea, we will have to make `n_tree` predictions, using only the subset of observations that for each tree is kept "out".

We will have then `n_tree` RMSE, and we can average those to have an averate RMSE of the OOB observations.

    n_tree = 50
    Rf_model <- randomForest(mpg ~., ntree = n_tree, data = mtcars, keep.inbag=T)  # we use keep.inbag = T
    
    inbag <- lapply(1:n_tree, function(x) which(Rf_model[["inbag"]][ ,x] == 0)) # we get only the "zeros"
    # to look inside use View(Rf_model[["inbag"]]), I think that the zeros are the OOB
    
    
    rf_pred <- lapply(inbag, function(x) predict(Rf_model, mtcars[x, ])) # predictions
    
    
    (oob_err <- map2_dbl(rf_pred, inbag, function(x, y) rmse_function(x, mtcars[y, ]$mpg)))
    # [1] 1.03926667 0.01556667 2.98096667 1.27210000 1.86380000 2.25883333 3.49130000 0.18763333 1.59326667 0.11236667
    # [11] 6.92163333 0.40183333 3.36586667 1.19960000 1.31833333 2.88373333 4.48326667 1.67406667 6.92566667 8.51793333
    # [21] 3.32893333 0.65510000 3.87440000 1.89276667 3.51290000 3.13026667 4.81453333 0.59756667 1.56783333 6.12180000
    # [31] 3.54490000 0.57406667 0.20236667 2.20220000 0.23226667 1.61360000 0.32690000 1.86300000 3.38393333 3.33723333
    # [41] 1.43760000 6.63860000 0.13120000 1.48580000 1.32950000 2.85310000 2.01306667 2.16363333 4.80706667 1.74310000
    
    mean(oob_err) # mean of the RMSEs
    #[1] 2.477725


  [1]: https://en.wikipedia.org/wiki/Root-mean-square_deviation