1
$\begingroup$

I'm building a classification model in R using random forest and the package caret. I'm interested in which parameters are optimised during resampling.

As an example, lets use the iris dataset and fit two models - one that uses no resampling, and one based on 10-fold cross validation:

set.seed(99)
mod1 <- train(Species ~., data = iris,
          method = "rf",
          ntree = 500,
          tuneGrid = data.frame(mtry=2),
          trControl = trainControl(method = "none"))


set.seed(99)
mod2 <- train(Species ~., data = iris,
          method = "rf",
          ntree = 500,
          tuneGrid = data.frame(mtry=2),
          trControl = trainControl(method = "repeatedcv", number=10,repeats=1))

As we can see, in both models the number of random predictors per split (mtry) is 2, and there are 500 trees generated. Obviously the two models give different results, but what are the parameters that are optimised during cross validaton?

As a comparison, Kuhn in his presentation talks about rpart (slides 52 - 69), where he explains that during resampling we actually prune the tree.

But what about when we're using random forest? Are the generated trees pruned as well, or there are other parameters that are optimised (e.g. max depth)?

$\endgroup$
2
$\begingroup$

You are not optimizing any parameters in your code. The only tuning parameter considered in the caret package is the mtry value, which is specified to be 2 in your code. However, it is still important to get a good estimate of the accuracy of the random forest; model 2 shows the accuracy is around 95.3% using repeated K-fold cross-validation. This is similar to what we get using the out-of-bag (OOB) sample estimate from the random forest:

randomForest(Species ~ ., data=iris, ntree=500, mtry=2)

Random forest does not prune the trees. I believe the only other parameter you may want to optimize in randomForest is the nodesize. This is set to 1 for classification, but Lin and Jeon (2006) found increasing the terminal node size may yield more accurate predictions. You'll need to tune this parameter yourself though (not a tuning parameter in caret package). There are also other tree-based models you can consider (e.g., Gradient Boosting Trees, Extremely Randomized Trees). You can see a list of the tuning parameters on the github page:

http://topepo.github.io/caret/train-models-by-tag.html#Random_Forest

$\endgroup$
  • $\begingroup$ Yes, optimal is not the best word here...So, if I understand correctly, by doing CV, we're only estimating the accuracy of the model and not influencing the efficiency of the model? This is a bit strange to me, because my experience (working on a similar dataset) is that results do get better by adding CV. And I think this doesn't coincide with the statement by Kuhn (check slide 46), especially since the other tuning parameters are in this case fixed. $\endgroup$ – Damjan Feb 19 '18 at 8:10
  • $\begingroup$ CV is used to get good estimates of the performance, which can be used to compare different models or parameter settings. You're not influencing the efficiency because there is only one model/parameter considered (you can see the code optimize the parameter if you take out the tuneGrid statement in mod2). Does slide 46 refer to "The Big Picture" slide? In your case, you only have one parameter set (mtry = 2). $\endgroup$ – Peter Calhoun Feb 19 '18 at 17:08
  • $\begingroup$ Tuning the sample size is also useful. See this paper: repositorio.uam.es/bitstream/handle/10486/664127/… $\endgroup$ – PhilippPro Feb 23 '18 at 15:18
1
$\begingroup$

As did @Peter emphasized, there is no tuning while using resampling (e.g. CV), at least not using the presented example.

But there's no doubt that there is difference in the predictions of the two models. One quick check is the following:

identical(predict(mod1,iris, type="prob"),predict(mod2,iris, type="prob"))

, which gives FALSE. To get a valid result in this code block, we have to redefine trainControl for mod1 and mod2:

trainControl(...,classProbs = TRUE, summaryFunction = multiClassSummary)

The reason why the model results differ is because the model in mod2 is aggregated based on all the folds used for CV, whereas for mod1 the model is fitted on the whole training data.

This is why the models give different results, and as @Peter pointed out, the model based on CV isn't more efficient than the one with no resampling.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.