I am training a random forest classifier in a setting with such a low sample size that I cannot afford setting aside validation and test sets. I train the hyperparameters via cross-validation and utilize the out-of-bag error of the final model (retrained on the whole dataset with the hyperparameters learned via CV) as an estimator for the generalization error when the model is applied to new data. Is this a sound procedure or is the OOB error overly optimistic, since it is based on the data which has also been used for training the hyperparameters?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Yes, in that case, the OOB error is a poor measure of performance.
In general, the final model evaluation should be performed on data which was not used for model training or tuning. This goes for all models, not just random forest.
-
$\begingroup$ Thank you. I guess this is a typical example of data leakage. I compared the results of the described approach with nested cross validation, using leave-one-out CV in the outer loop and the original CV in the inner loop, which should provide an unbiased estimate. The results are considerably worse using this approach. $\endgroup$ Apr 11, 2019 at 7:27