I keep getting different out-of-bag error estimates from the caret package, depending on how the estimates are computed. I can't seem to nail down exactly where the discrepancy is coming from.

Consider a simple random forest model using the built-in mtcars dataset:

tc <- caret::trainControl( method="oob" )
m <- caret::train( mpg ~ ., data=mtcars, method="rf", trControl=tc )
# Random Forest 

# 32 samples
# 10 predictors

# No pre-processing
# Resampling results across tuning parameters:

#   mtry  RMSE      Rsquared 
#    2    2.429726  0.8322324
#    6    2.301918  0.8494180
#   10    2.429676  0.8322394

# RMSE was used to select the optimal model using  the smallest value.
# The final value used for the model was mtry = 6.

A print out of the model reports that mtry = 6 was the best meta-parameter value and the associated error estimates are RMSE = 2.301918 and R^2 = 0.8494180. My understanding is that these error estimates are computed over out-of-bag examples in the data due to trainControl( method="oob" ) setting. Is this correct?

If I now apply the oob() function from $modelInfo to the best model (as captured by $finalModel), I would expect to get the same performance estimates as the mtry=6 entry in the table above, but I don't:

m$modelInfo$oob( m$finalModel )
#      RMSE  Rsquared 
# 2.3485040 0.8432614 

Furthermore, if I now compute predict( m$finalModel ), it should return out-of-bag predictions, since newdata parameter is omitted. However, this results in yet another estimate of OOB error when passed to postResample:

caret::postResample( predict(m$finalModel), mtcars$mpg )
#      RMSE  Rsquared       MAE 
# 2.3485040 0.8468821 1.8463368 

(Notice that although RMSE agrees with the oob() call above, R^2 has a different value.)

While I appreciate that the values are in the same "ballpark", I don't understand where the discrepancy comes from. Does anybody have any insight?


closed as off-topic by Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, Andy Dec 20 '17 at 10:24

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, Andy
If this question can be reworded to fit the rules in the help center, please edit the question.


It seems to be impossible to get to the tuning resamples if you use oob as a method, unless you write a custom method that saves away the fit objects during tuning.

  # Copy all model structure info from existing model type
  cust.mdl <- getModelInfo("rf", regex=FALSE)[[1]]

  # Override fit function so that we can save the iteration
  cust.mdl$fit <- function(x=x, y=y, wts=wts, param=param, lev=lev, last=last, classProbs=classProbs, ...) {
    # Dont save the final pass (dont train the final model across the entire training set)
    if(last == TRUE) return(NULL) 
    # Fit the model
    fit.obj <- getModelInfo("rf", regex=FALSE)[[1]]$fit(x, y, wts, param, lev, last, classProbs, ...)

    # Create an object with data to save and save it
    fit.data <- list(resample=rownames(x),
                   #x, y, wts,
                   param=param, lev=lev, last=last, classProbs=classProbs, 

    # Create a string representing the tuning params
    param.str <- paste(lapply(1:ncol(param), function(x) {
                       paste0(names(param)[x], param[1,x])
                       }), collapse="-")

    save(fit.data, file=paste0("rf_model_", param.str, ".RData"))
  return (fit.obj)

tc <- trainControl( method="oob")
m <- train( mpg ~ ., data=mtcars, method=cust.mdl, trControl=tc)

The final fit object returns oob errors from each tuning iteration

         RMSE  Rsquared mtry
    1 2.417379 0.8339331    2
    2 2.394624 0.8370448    6
    3 2.455534 0.8286495   1

So now if I fetch the fit object from the tuning iteration corresponding to mtry=6, I can recreate those errors


# using same oob() function that is used for final predictions
     RMSE  Rsquared 
2.3946242 0.8370448 
  • $\begingroup$ Thanks for this. I think I get it: the discrepancy comes from the fact that different data resampling is used for tuning the parameter grid as opposed to training the final model. I notice that using your approach still produces a different value for caret::R2( predict(fit.data$mdl), mtcars$mpg ). Any thoughts on why that might be happening? (The value from caret::RMSE( predict(fit.data$mdl), mtcars$mpg ) matches exactly.) $\endgroup$ – Artem Sokolov Dec 19 '17 at 15:30
  • $\begingroup$ I believe rsq is calculated using traditional formula and not corr.squared. Use caret::R2(pred, obs, formula = "traditional") if you want to recreate rsquared in resampling object $\endgroup$ – dmi3kno Dec 19 '17 at 17:17
  • 1
    $\begingroup$ Ah, yep. I see it now: caret::R2( predict(fit.data$mdl), mtcars$mpg, formula = "traditional" ) gives the same value. $\endgroup$ – Artem Sokolov Dec 19 '17 at 17:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.