# Inconsistent Out-of-bag error estimates [closed]

I keep getting different out-of-bag error estimates from the caret package, depending on how the estimates are computed. I can't seem to nail down exactly where the discrepancy is coming from.

Consider a simple random forest model using the built-in mtcars dataset:

set.seed(100)
tc <- caret::trainControl( method="oob" )
m <- caret::train( mpg ~ ., data=mtcars, method="rf", trControl=tc )
# Random Forest

# 32 samples
# 10 predictors

# No pre-processing
# Resampling results across tuning parameters:

#   mtry  RMSE      Rsquared
#    2    2.429726  0.8322324
#    6    2.301918  0.8494180
#   10    2.429676  0.8322394

# RMSE was used to select the optimal model using  the smallest value.
# The final value used for the model was mtry = 6.


A print out of the model reports that mtry = 6 was the best meta-parameter value and the associated error estimates are RMSE = 2.301918 and R^2 = 0.8494180. My understanding is that these error estimates are computed over out-of-bag examples in the data due to trainControl( method="oob" ) setting. Is this correct?

If I now apply the oob() function from $modelInfo to the best model (as captured by $finalModel), I would expect to get the same performance estimates as the mtry=6 entry in the table above, but I don't:

m$modelInfo$oob( m$finalModel ) # RMSE Rsquared # 2.3485040 0.8432614  Furthermore, if I now compute predict( m$finalModel ), it should return out-of-bag predictions, since newdata parameter is omitted. However, this results in yet another estimate of OOB error when passed to postResample:

caret::postResample( predict(m$finalModel), mtcars$mpg )
#      RMSE  Rsquared       MAE
# 2.3485040 0.8468821 1.8463368


(Notice that although RMSE agrees with the oob() call above, R^2 has a different value.)

While I appreciate that the values are in the same "ballpark", I don't understand where the discrepancy comes from. Does anybody have any insight?

It seems to be impossible to get to the tuning resamples if you use oob as a method, unless you write a custom method that saves away the fit objects during tuning.

  library(caret)
# Copy all model structure info from existing model type
cust.mdl <- getModelInfo("rf", regex=FALSE)[[1]]

# Override fit function so that we can save the iteration
cust.mdl$fit <- function(x=x, y=y, wts=wts, param=param, lev=lev, last=last, classProbs=classProbs, ...) { # Dont save the final pass (dont train the final model across the entire training set) if(last == TRUE) return(NULL) # Fit the model fit.obj <- getModelInfo("rf", regex=FALSE)[[1]]$fit(x, y, wts, param, lev, last, classProbs, ...)

# Create an object with data to save and save it
fit.data <- list(resample=rownames(x),
mdl=fit.obj,
#x, y, wts,
param=param, lev=lev, last=last, classProbs=classProbs,
other=list(...))

# Create a string representing the tuning params
param.str <- paste(lapply(1:ncol(param), function(x) {
paste0(names(param)[x], param[1,x])
}), collapse="-")

save(fit.data, file=paste0("rf_model_", param.str, ".RData"))
return (fit.obj)
}

set.seed(100)
tc <- trainControl( method="oob")
m <- train( mpg ~ ., data=mtcars, method=cust.mdl, trControl=tc)


The final fit object returns oob errors from each tuning iteration

m$results > RMSE Rsquared mtry 1 2.417379 0.8339331 2 2 2.394624 0.8370448 6 3 2.455534 0.8286495 1  So now if I fetch the fit object from the tuning iteration corresponding to mtry=6, I can recreate those errors load("rf_model_mtry6.RData") # using same oob() function that is used for final predictions m$modelInfo$oob(fit.data$mdl)
>
RMSE  Rsquared
2.3946242 0.8370448

• Thanks for this. I think I get it: the discrepancy comes from the fact that different data resampling is used for tuning the parameter grid as opposed to training the final model. I notice that using your approach still produces a different value for caret::R2( predict(fit.data$mdl), mtcars$mpg ). Any thoughts on why that might be happening? (The value from caret::RMSE( predict(fit.data$mdl), mtcars$mpg ) matches exactly.) – Artem Sokolov Dec 19 '17 at 15:30
• I believe rsq is calculated using traditional formula and not corr.squared. Use caret::R2(pred, obs, formula = "traditional") if you want to recreate rsquared in resampling object – dmi3kno Dec 19 '17 at 17:17
• Ah, yep. I see it now: caret::R2( predict(fit.data$mdl), mtcars$mpg, formula = "traditional" ) gives the same value. – Artem Sokolov Dec 19 '17 at 17:23