0
$\begingroup$

I keep getting different out-of-bag error estimates from the caret package, depending on how the estimates are computed. I can't seem to nail down exactly where the discrepancy is coming from.

Consider a simple random forest model using the built-in mtcars dataset:

set.seed(100)
tc <- caret::trainControl( method="oob" )
m <- caret::train( mpg ~ ., data=mtcars, method="rf", trControl=tc )
# Random Forest 

# 32 samples
# 10 predictors

# No pre-processing
# Resampling results across tuning parameters:

#   mtry  RMSE      Rsquared 
#    2    2.429726  0.8322324
#    6    2.301918  0.8494180
#   10    2.429676  0.8322394

# RMSE was used to select the optimal model using  the smallest value.
# The final value used for the model was mtry = 6.

A print out of the model reports that mtry = 6 was the best meta-parameter value and the associated error estimates are RMSE = 2.301918 and R^2 = 0.8494180. My understanding is that these error estimates are computed over out-of-bag examples in the data due to trainControl( method="oob" ) setting. Is this correct?

If I now apply the oob() function from $modelInfo to the best model (as captured by $finalModel), I would expect to get the same performance estimates as the mtry=6 entry in the table above, but I don't:

m$modelInfo$oob( m$finalModel )
#      RMSE  Rsquared 
# 2.3485040 0.8432614 

Furthermore, if I now compute predict( m$finalModel ), it should return out-of-bag predictions, since newdata parameter is omitted. However, this results in yet another estimate of OOB error when passed to postResample:

caret::postResample( predict(m$finalModel), mtcars$mpg )
#      RMSE  Rsquared       MAE 
# 2.3485040 0.8468821 1.8463368 

(Notice that although RMSE agrees with the oob() call above, R^2 has a different value.)

While I appreciate that the values are in the same "ballpark", I don't understand where the discrepancy comes from. Does anybody have any insight?

$\endgroup$

closed as off-topic by Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, Andy Dec 20 '17 at 10:24

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, Andy
If this question can be reworded to fit the rules in the help center, please edit the question.

2
$\begingroup$

It seems to be impossible to get to the tuning resamples if you use oob as a method, unless you write a custom method that saves away the fit objects during tuning.

  library(caret)
  # Copy all model structure info from existing model type
  cust.mdl <- getModelInfo("rf", regex=FALSE)[[1]]

  # Override fit function so that we can save the iteration
  cust.mdl$fit <- function(x=x, y=y, wts=wts, param=param, lev=lev, last=last, classProbs=classProbs, ...) {
    # Dont save the final pass (dont train the final model across the entire training set)
    if(last == TRUE) return(NULL) 
    # Fit the model
    fit.obj <- getModelInfo("rf", regex=FALSE)[[1]]$fit(x, y, wts, param, lev, last, classProbs, ...)

    # Create an object with data to save and save it
    fit.data <- list(resample=rownames(x),
                   mdl=fit.obj,
                   #x, y, wts,
                   param=param, lev=lev, last=last, classProbs=classProbs, 
                   other=list(...))

    # Create a string representing the tuning params
    param.str <- paste(lapply(1:ncol(param), function(x) {
                       paste0(names(param)[x], param[1,x])
                       }), collapse="-")

    save(fit.data, file=paste0("rf_model_", param.str, ".RData"))
  return (fit.obj)
}

set.seed(100)
tc <- trainControl( method="oob")
m <- train( mpg ~ ., data=mtcars, method=cust.mdl, trControl=tc)

The final fit object returns oob errors from each tuning iteration

m$results
>
         RMSE  Rsquared mtry
    1 2.417379 0.8339331    2
    2 2.394624 0.8370448    6
    3 2.455534 0.8286495   1

So now if I fetch the fit object from the tuning iteration corresponding to mtry=6, I can recreate those errors

load("rf_model_mtry6.RData")

# using same oob() function that is used for final predictions
m$modelInfo$oob(fit.data$mdl)
>
     RMSE  Rsquared 
2.3946242 0.8370448 
$\endgroup$
  • $\begingroup$ Thanks for this. I think I get it: the discrepancy comes from the fact that different data resampling is used for tuning the parameter grid as opposed to training the final model. I notice that using your approach still produces a different value for caret::R2( predict(fit.data$mdl), mtcars$mpg ). Any thoughts on why that might be happening? (The value from caret::RMSE( predict(fit.data$mdl), mtcars$mpg ) matches exactly.) $\endgroup$ – Artem Sokolov Dec 19 '17 at 15:30
  • $\begingroup$ I believe rsq is calculated using traditional formula and not corr.squared. Use caret::R2(pred, obs, formula = "traditional") if you want to recreate rsquared in resampling object $\endgroup$ – dmi3kno Dec 19 '17 at 17:17
  • 1
    $\begingroup$ Ah, yep. I see it now: caret::R2( predict(fit.data$mdl), mtcars$mpg, formula = "traditional" ) gives the same value. $\endgroup$ – Artem Sokolov Dec 19 '17 at 17:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.