22
$\begingroup$

This question already has an answer here:

I am puzzled as to why the caret package in R does not allow tuning on the number of trees (ntree) in a random forest (specifically in the randomForest package)? I cant imagine this is an oversight on the part of the package author - so there must be a reason for it? Can anyone shed light?

$\endgroup$

marked as duplicate by Tim Apr 17 at 20:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

16
$\begingroup$

In theory, the performance of a RF model should be a monotonic function of ntree that plateaus beyond a certain point once you have 'enough' trees. This makes ntree more of a performance parameter than a Goldilocks parameter that you would want to tune. Caret tends to focus on tuning parameters that perform poorly for high and low values in which you want to find the happy medium.

In practice I believe there may have been studies that have found performance does reduce for very large ntree values but even if this is true the effect is subtle and requires very large forests.

There are at least 2-3 other parameters to RF that Caret doesn't tune for the same reasons as ntree.

$\endgroup$
  • $\begingroup$ So, how do we know how many trees are used? $\endgroup$ – Hack-R Mar 13 '15 at 13:49
  • $\begingroup$ If you print the model, it tells you how many trees it used, which i think is fixed to 500. $\endgroup$ – Asymptote Mar 25 '15 at 11:09
  • 2
    $\begingroup$ Accuracy is half the question. Engineering says "fast, cheap, or good: pick two". So look at improvement in accuracy vs. compute time. There should be a best num.trees after which adding another is cost with no benefit. $\endgroup$ – EngrStudent May 29 '17 at 17:59
9
$\begingroup$

Caret does let you tune the number of trees on its backend randomForest package. For instance, considering the latest version (4.6-12) as of now, you just pass the normal ntree parameter. caret will "repass" it to randomForest, e.g.:

train(formula,
      data = mydata,
      method = "rf",
      ntree = 5,
      trControl = myTrControl)
$\endgroup$
  • 6
    $\begingroup$ it seems like you can pass the ntree parameter to train() but you cannot use it in a tuning grid via the argument tuneGrid $\endgroup$ – roman Oct 31 '16 at 15:03
  • 1
    $\begingroup$ Why would you purposefully decrease functionality when it would be so easy to implement? pull request coming up... $\endgroup$ – wordsforthewise Jan 31 '18 at 3:00
  • $\begingroup$ Because it's statistically incorrect to do so. $\endgroup$ – Matthew Drury Apr 17 at 20:43
  • $\begingroup$ Sure, we don't need to tune the number of trees (just set it to 500 or so). But tuning other things like maxnodes and/or nodesize can be useful, especially if the RF is overfitting. $\endgroup$ – wordsforthewise May 22 at 17:20
1
$\begingroup$

If you already have an idea about how many trees you want to use (Breiman recommends at least 1000) and have used randomForest::tuneRF to get an optimal mtry value (let's use 6 as an example), then:

ctrl <- trainControl(method = "none")

set.seed(2)
rforest <- train(response ~ ., data = data_set,
               method = "rf",
               ntree = 1000,
               trControl = ctrl,
               tuneGrid = data.frame(mtry = 6))

Eduardo has answered your question above but I wanted to additionally demonstrate how you can tune the value for the number of random variables used for partitioning. When tuning a random forest, this parameter has more importance than ntree as long as ntree is sufficiently large.

$\endgroup$
  • $\begingroup$ This answer appears to be essentially a repetition of the existing answer from Eduardo. $\endgroup$ – mkt Jul 12 '18 at 8:42
  • 2
    $\begingroup$ It is clearly not a repetition. I have shown how to use tuneGrid as well as demonstrating how to use trControl. I believe my example is clearer as it shows that you must explicitly specify the number of trees while you can tune the number of random variables to partition on. $\endgroup$ – Seanosapien Jul 12 '18 at 10:11
  • $\begingroup$ Agreed that tuneGrid and trControl are new, at least partly. But neither of those address the question, which is about ntree. The relevant part of the answer is the same as Eduardo's. But if you believe they are important, I suggest you highlight that while also explaining how it relates to the question. $\endgroup$ – mkt Jul 12 '18 at 11:19
0
$\begingroup$

Though I agree the with the theoretical explanations posted here, in practice, having a too large number of trees is a waste of computational power and makes the model objects uncomfortably heavy for working with them (especially if you use to constantly save and load .RDS objects). Because of that, I think if we want models to be adequate we have to find somehow the minimum necessary number of trees that allow for a stable performance (and then "let the asymptotic behavior of LLN do the rest"). Perhaps if you are a very experienced statistician or if you are always working on similar problems you can use some rule of thumb (say 1000 or 10000 trees). But if your work requires you to adapt to a variety of modelling tasks, you'll end up needing some calibration method that allows you for finding an adequate and inexpensive number of trees.

For this purpose, you could just download the source code of the method from here and then rewrite it to create a custom method that adapts to your needs. Feel free to use the following example:

customRF <- list(label = "Random Forest",
 library = "randomForest",
 loop = NULL,
 type = c("Classification", "Regression"),
 parameters = data.frame(parameter = c("mtry", "ntree"), class = rep("numeric", 2), label = c("mtry", "ntree")),
 grid = function(x, y, len = NULL, search = "grid") {
   if(search == "grid") {
     out <- data.frame(mtry = caret::var_seq(p = ncol(x), 
                                             classification = is.factor(y), 
                                             len = len))
   } else {
     out <- data.frame(mtry = unique(sample(1:ncol(x), size = len, replace = TRUE)))
   }
   out
 },
 fit = function(x, y, wts, param, lev, last, classProbs, ...)
   randomForest::randomForest(x, y, mtry = param$mtry, ntree=param$ntree...),
 predict = function(modelFit, newdata, submodels = NULL)
   if(!is.null(newdata)) predict(modelFit, newdata) else predict(modelFit),
 prob = function(modelFit, newdata, submodels = NULL)
   if(!is.null(newdata)) predict(modelFit, newdata, type = "prob") else predict(modelFit, type = "prob"),
 predictors = function(x, ...) {
   ## After doing some testing, it looks like randomForest
   ## will only try to split on plain main effects (instead
   ## of interactions or terms like I(x^2).
   varIndex <- as.numeric(names(table(x$forest$bestvar)))
   varIndex <- varIndex[varIndex > 0]
   varsUsed <- names(x$forest$ncat)[varIndex]
   varsUsed
 },
 varImp = function(object, ...){
   varImp <- randomForest::importance(object, ...)
   if(object$type == "regression")
     varImp <- data.frame(Overall = varImp[,"%IncMSE"])
   else {
     retainNames <- levels(object$y)
     if(all(retainNames %in% colnames(varImp))) {
       varImp <- varImp[, retainNames]
     } else {
       varImp <- data.frame(Overall = varImp[,1])
     }
   }

   out <- as.data.frame(varImp)
   if(dim(out)[2] == 2) {
     tmp <- apply(out, 1, mean)
     out[,1] <- out[,2] <- tmp  
   }
   out
 },
 levels = function(x) x$classes,
 tags = c("Random Forest", "Ensemble Model", "Bagging", "Implicit Feature Selection"),
 sort = function(x) x[order(x[,1]),],
 oob = function(x) {
   out <- switch(x$type,
                 regression =   c(sqrt(max(x$mse[length(x$mse)], 0)), x$rsq[length(x$rsq)]),
                 classification =  c(1 - x$err.rate[x$ntree, "OOB"],
                                     e1071::classAgreement(x$confusion[,-dim(x$confusion)[2]])[["kappa"]]))
   names(out) <- if(x$type == "regression") c("RMSE", "Rsquared") else c("Accuracy", "Kappa")
   out
 })

After defining this custom method you only have to call it from train(method=customRF) and both mtry and ntree will be calibrated depending on your trainControl() specifications.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.