1
$\begingroup$

I'm using machine learning methods in R for descriptive regression modelling of a small dataset. I have fit random forest (randomForest), unbiased random forest (cforest) and boosted regression trees (gbm) using caret with 10 fold cross-validation and 5 repeats.

I've tuned the hyperparameters for each model and achieved the same cross-validation RMSE accuracy. However, only the random forest model gives a satisfactory fit for the training data. Below are the calibration plots and fit stats (the red, horizontal line in each is the observed mean corresponding to the null RMSE).

I have used predict() on the caret models and am confident that these are predictions are for the training set. Predictor variable importance statistics and partial dependence plots are qualitatively similar for all three models (except that cforest does more to reduce bias). So, while these three models detected similar patterns in the data, how come two of the three fit poorly to the training data? My choice was random forest, but I haven't found an effective method for correcting for bias in the importance statistics for that model.

enter image description here
enter image description here enter image description here

UPDATE: The following code and plots provide examples using the iris dataset. Note that, like my data above, randomForest RMSEtrain is consistently c. 0.6 times RMSEtrain for cforest and gbm models. I think randomForest is overfitting and therefore comparing calibration plots and RMSEtrain among different models is misleading?

    data(iris)
    
    ## models without added noise
    
    # null model
    RMSEnull <- sqrt(mean((iris[, 1] - mean(iris[, 1]))^2))
    
    # random forest
    set.seed(69)
    mygrid <- data.frame(mtry=c(1, 2, 3))
    mycontrol <- trainControl(method="repeatedcv", number=5, 
                      repeats=5)
    mymod.rf <- train(x=iris[, 2:4], y=iris[, 1], method="rf", 
                    trControl=mycontrol, tuneGrid=mygrid)
    round(head(mymod.rf$results[order(mymod.rf$results$RMSE), ]), 3)
RMSEholdout <- mymod.rf$results[order(mymod.rf$results$RMSE), ]
    RMSEholdout <- RMSEholdout[1, 2]
    RMSEtrain <- sqrt(mean((iris[, 1] - predict(mymod.rf))^2))
    plot(iris[,1], predict(mymod.rf), xlim=c(4,8), ylim=c(4,8), 
           xlab="observed", ylab="randomForest")
    abline(h=mean(iris[,1]), col="red")
    abline(0,1, lty="dotted")
    text(4.5, 7.5, paste("RMSEnull =", round(RMSEnull, 3)), 
             pos=4, col="red")
    text(4.5, 7.0, paste("RMSEholdout =", round(RMSEholdout,3)), 
           pos=4)
    text(4.5, 6.5, paste("RMSEtrain =", round(RMSEtrain, 3)), 
          pos=4)
    
    # conditional random forest unbiased
    set.seed(69)
    mygrid <- data.frame(mtry=c(1, 2, 3))
    mycontrol <- trainControl(method="repeatedcv", number=5, 
            repeats=5)
    mymod.cf <- train(x=iris[,2:4], y=iris[,1], method="cforest", 
                    trControl<-mycontrol, tuneGrid=mygrid)
 
    round(head(mymod.cf$results[order(mymod.cf$results$RMSE), ]), 
3)
RMSEholdout <- mymod.cf$results[order(mymod.cf$results$RMSE),]
    RMSEholdout <- RMSEholdout[1, 2]
    RMSEtrain <-  sqrt(mean((iris[, 1] - predict(mymod.cf))^2))
    plot(iris[,1], predict(mymod.cf), xlim=c(4,8), ylim=c(4,8), 
         xlab="observed", ylab="cforest_unbiased")
    abline(h=mean(iris[, 1]), col="red")
    abline(0,1, lty="dotted")
    text(4.5, 7.5, paste("RMSEnull =", round(RMSEnull, 3)), 
          pos=4, col="red")
    text(4.5, 7.0, paste("RMSEholdout =", round(RMSEholdout,3)), 
          pos=4)
    text(4.5, 6.5, paste("RMSEtrain =", round(RMSEtrain, 3)), 
         pos=4)
    
    # boosted regression tree
    set.seed(69)
    mygrid <- expand.grid(shrinkage=0.01, interaction.depth=c(1, 
            2), n.minobsinnode=c(5, 10, 20), 
             n.trees=seq(100, 2000, 100))
    mycontrol <- trainControl(method="repeatedcv", number=5, 
               repeats=5)
    mymod.gbm <- train(x=iris[, 2:4], y=iris[, 1], method="gbm", 
               trControl=mycontrol, tuneGrid=mygrid, verbose = F)
    round(head(mymod.gbm$results[order(mymod.gbm$results$RMSE), 
              ]), 3)
RMSEholdout <- 
  mymod.gbm$results[order(mymod.gbm$results$RMSE), ]
    RMSEholdout <- RMSEholdout[1, 5]
    RMSEtrain <- sqrt(mean((iris[, 1] - predict(mymod.gbm))^2))
    plot(iris[, 1], predict(mymod.gbm), xlim=c(4,8), ylim=c(4, 8), 
           xlab="observed", ylab="gbm")
    abline(h=mean(iris[,1]), col="red")
    abline(0,1, lty="dotted")
    text(4.5, 7.5, paste("RMSEnull =", round(RMSEnull, 3)), 
         pos=4, col="red")
    text(4.5, 7.0, paste("RMSEholdout =", round(RMSEholdout,3)), 
            pos=4)
    text(4.5, 6.5, paste("RMSEtrain =", round(RMSEtrain, 3)), 
         pos=4)
    
    ## models with added noise
    # this should do
    plot(iris[,1], iris[,1]+rnorm(dim(iris)[1], 0, 0.4))
    noisyiris <- [, 1] + rnorm(dim(iris)[1], 0, 0.4)
    
    # and the remaining code is like above, 
    #with noisy replacing iris[,1]
    # I'm not going to present the additional three plots

enter image description here enter image description here enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you mean two out of the three fit poorly to the data? The last two have the same RMSEtrain of 0.364 which is still higher than the null model, with randomForest having the smallest error. Have you tried running the models a couple more times to check if you get similar results? $\endgroup$
    – user2974951
    Commented Jan 18, 2019 at 12:29
  • $\begingroup$ There's a gross difference in scatterplots and RMSE for the training data. My questions are how this occurs (considering that CV RMSEs are similar) and whether that's important? E.g. Leathwick et al. (2006) reported similar residual deviance for training and holdout data, but Oppel et al. (2012) reported calibration plots and results. Refs: Leathwick et al. (2006). Variation in ... fish species richness ... using boosted regression trees. Mar Ecol Prog Ser 321: 267-281. Oppel, S. et al. (2012). Comparison of five ... techniques to predict ... seabirds. Biol Cons 156: 94–104. $\endgroup$
    – stweb
    Commented Jan 19, 2019 at 7:53

2 Answers 2

Reset to default
2
$\begingroup$

Random forests, and some other machine learning techniques, are notorious for failing to achieve good absolute predictive accuracy (calibration curve = line of identity). I think of unbiased calibration assessment (using resampling or, if the sample size is huge, using external validation) as a test that a predictive method must passed in order to be a good candidate, then other attributes are assessed such as predictive discrimination. I don't tend to compare two calibration curves, but to only use methods achieving good absolute accuracy.

Improve this answer
$\endgroup$
1
  • $\begingroup$ In the end, regression and multi-model inference were more useful for my study. Machine learning methods could be useful for larger datasets. $\endgroup$
    – stweb
    Commented Jan 17, 2022 at 1:11
0
$\begingroup$

I don't know why randomForest has smaller RMSE on the training set, I would assume it's because RF by default builds deep trees with no pruning, hence why they may explain the training set a little better during building. However, in most cases this should not be a problem, it is hard to overfit a randomForest model, because of the many trees which will, on average, even things out. Cforest and gmb on the other hand have by default set limits on the depth of the trees and on-the-fly pruning, and this is why I think they have worse performance on the training set. Nonetheless, all models have very similar RMSE based on CV so they all perform equally well. Choosing a model now is just a matter of preference.

Improve this answer
$\endgroup$
4
  • $\begingroup$ OK, this post, suggests that, for random forests in particular, performance on training set data is meaningless. It appears to me that calibration plots are only for 'explanatory modelling' (Shmueli 2010) where the theory is well defined and goodness of fit (i.e. to the training data) is informative. That random forests fit the training data better than other ML models is a trap. Ref: Shmueli, G. (2010). To explain or to predict?. Statistical Science, 25(3), 289-310. $\endgroup$
    – stweb
    Commented Jan 19, 2019 at 22:43
  • $\begingroup$ @stweb So it is as I said, the randomForest package grows very deep trees, so they perform very well on training data (especially classification). Also as noted, this is true for almost any model and you should not rely on this measure, and you should use the OOB / CV approach for an estimate of error. $\endgroup$
    – user2974951
    Commented Jan 20, 2019 at 8:27
  • $\begingroup$ I have also found that I can make cforest fit the training data by reducing the parameters minsplit and minbucket. Sooner or later we might read a paper on 'Use and misuse of calibration plots' (in Ecology, at least). $\endgroup$
    – stweb
    Commented Jan 21, 2019 at 12:19
  • $\begingroup$ @stweb I don't know specifically about gbm, but I know that boosting also allows for building deep trees, so it is not specifically isolated to the other 2 methods. I don't know about ecology, but in the machine learning / statistics world no one takes training error seriously, since it is known to be a poor performance metric (optimistic). You look only at CV error or a separate test set prediction error. $\endgroup$
    – user2974951
    Commented Jan 21, 2019 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.