I would like to tune the hyperparameters of a random forest and then obtain an unbiased score of its performance.
As I understand, the natural way would be to use nested cross validation. However, it seems to me that with this method the best hyperparameters found in each inner loop might differ across loops, which then creates a problem when you want to report the hyperparameter settings for which the average score was obtained. Ideally, I would like to first settle on a hyperparameter setting, then get an error measure specific to that setting.
I came up with the following procedure:
- Separate the dataset into a validation and a train+test set.
- Perform a grid search using cross validation on the validation set to find optimal hyperparameterst.
- Fit the random forest with the optimal hyperparameters on the train+test set, and report the out-of-bag error.
Point 3. rests on the fact that cross validation is essentially not needed with random forests, as the out-of-bag error is unbiased.
I would like to make sure that this is a sound approach, or whether perhaps there is a more natural way of doing this.