I would like to tune the hyperparameters of a random forest and then obtain an unbiased score of its performance.

As I understand, the natural way would be to use nested cross validation. However, it seems to me that with this method the best hyperparameters found in each inner loop might differ across loops, which then creates a problem when you want to report the hyperparameter settings for which the average score was obtained. Ideally, I would like to first settle on a hyperparameter setting, then get an error measure specific to that setting.

I came up with the following procedure:

  1. Separate the dataset into a validation and a train+test set.
  2. Perform a grid search using cross validation on the validation set to find optimal hyperparameterst.
  3. Fit the random forest with the optimal hyperparameters on the train+test set, and report the out-of-bag error.

Point 3. rests on the fact that cross validation is essentially not needed with random forests, as the out-of-bag error is unbiased.

I would like to make sure that this is a sound approach, or whether perhaps there is a more natural way of doing this.

  • 1
    $\begingroup$ You may want to define hyperparameters in the context of RF to clarify your question. If you imply n trees, n nodes, mtry, perhaps this would help stackoverflow.com/questions/13956435/… And I've also found limited improvement with most tuning as long as it is enough to converge. $\endgroup$
    – katya
    Mar 28, 2017 at 22:40

2 Answers 2


I think you got it quite right but not exactly. Here my suggestion:

  1. Separate the dataset into a test and a train+validation set.
  2. Perform a grid search using cross validation on the train set to find optimal hyperparameters optimized on the validation set (for random forest, this would be defining your mtry).
  3. Use the entire train+validation with the optimal hyperparameters and report the error using the test set.

Only this way, you ensure that the performance is measured on a part of the data the model has never seen. I recommend splitting Nr. 1 a few times, for example with the 10-fold CV, to make the performance measure less prone to variance.

The mlr package has good explanations on this nested cross-validation https://mlr-org.github.io/mlr-tutorial/devel/html/nested_resampling/index.html

  • 3
    $\begingroup$ Seems you're mixing up validation and test sets; the validation set is used for model selection/hyperparameter tuning, and the test set is used to measure/report performance. Unlike many models, random forests give an unbiased estimate of the error in the form of the out-of-bag error. This means that nested cross validation isn't necessary, saving a lot of computation. $\endgroup$
    – user20160
    Aug 1, 2017 at 15:19
  • $\begingroup$ I tried to use the names that the question had in, but I will change it if this is the default use of the names 'validation' and 'test'. Thank you. It is also good to know that oob should be unbiased, however, if you want a confidence interval for your prediction performance, for example, it might still be good to do an outer CV. What do you think? $\endgroup$
    – Brigitte
    Aug 1, 2017 at 15:45

You use nested cross-validation just to get an unbiased performance measure and use that choose an algorithm among a set of algorithms. After choosing an algorithm, you use the whole dataset with k-fold cv to tune your hyper-parameters specific to that algorithm. You use this final model to apply on data that you have never seen before. So, for performance measure -> neseted cv, for the final hyper-parameter tuning -> k-fold cv


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.