1
$\begingroup$

I have a question concerning OOB error in random forests and data partitioning. As far as i know in random forests the trees are not pruned. Also we use OOB error for measuring the performance of the forest. Why then we should use data partitioning (training - validation) when constructing a random forest. In many cases that i have seen a data partitioning process is used. In this case how can the validation error be interpreted?

Thanks in advance,

Andreas

$\endgroup$
4
$\begingroup$

Training a model, tuning its hyperparameters, and evaluating its performance are typically done using independent training, validation, and test sets. This three-way split can take the form of holdout or nested cross validation. The independence of these sets is important because, otherwise, estimates of the error would be downwardly biased--we'd select poor models and expect them to perform better on future data than they really would. Because random forests already use bootstrapping for fitting individual tries, they readily yield the out-of-bag (OOB) error. This is an unbiased estimate of the error on future data. As such, it can take the place of the validation or test error, and is cheaper to compute than using nested cross validation.

If we had a fixed set of hyperparameters, we could train a random forest on the entire dataset, estimate performance using the OOB error, and call it a day. But, random forests have hyperparameters that may need to be tuned to balance between under- and overfitting. One of these is the number of features considered for each split. Another is tree size, which is typically controlled by limiting the depth or number of nodes when growing the tree, rather than by pruning after the fact. Rather than splitting the data into training, validation, and test sets, we can use the OOB error in place of the the validation or test set error. For example, hyperparameters could be tuned to minimize OOB error and performance could be evaluated on the test set (possibly using cross validation, with no need for nesting).

$\endgroup$
  • $\begingroup$ Hello,Thanks for your answer. So you say that when using random forests there is no need for partitioning in training and validation; the oob set takes the place of the validation data set? As i told you i have seen cases where in random forests a data partitioning of the input data set takes place and then we have training error, validation error and OOB error. Why is that? $\endgroup$ – Andreas Zaras Aug 3 '17 at 8:15
  • $\begingroup$ OOB error could take the place of validation or test set error. In the case you mention, it sounds like it's the latter. So, the data are split into training and validation sets, using holdout or cross validation. The validation set is used to tune hyperparameters, and the OOB error is used to measure performance. $\endgroup$ – user20160 Aug 3 '17 at 9:25
  • $\begingroup$ Hello. Thanks for your answer. That makes sense: The validation set is used to tune hyperparameters and the OOB error is used to measure performance. $\endgroup$ – Andreas Zaras Aug 3 '17 at 11:04
  • $\begingroup$ But can you elaborate on the tuning of the hyperparameters. As far as i have read the hyperparameteres are set fixed before the creation of the random forest. For example in SAS HP Forest we choose the proportion of observations to bootstrap (default proportion=60%) we choose the number of inputs considered for each split (default = sqrt(number of inputs) etc. So the hyperparameteres are fixed. So what does fine tune the hyperparameters mean? Thanks in advance $\endgroup$ – Andreas Zaras Aug 3 '17 at 11:08
  • $\begingroup$ Tuning the hyperparameters means adjusting them to optimize the model's generalization performance (e.g. as estimated using the validation set). Standard software for random forests (and other models) may provide some reasonable default values. If you use these (or some other fixed values), then no tuning is taking place and you can skip using a validation set. But, tuning the hyperparameters can typically increase performance because the optimal values values depend on the data. $\endgroup$ – user20160 Aug 3 '17 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.