0
$\begingroup$

I have a following quiz:

A random forest is used for classifying the disease state of patients based on measuring multiple genes. The dataset consists of 100 genes and 50 patients. However, for each patient, 10 measurements exist at different time points.

When analyzing the data, the out-of-bag error is very close to the error on the training data, but very different from the error on the test dataset.

I am not quite sure if I understand it correctly. The out-of-bag error is calculated on the points (patients) that were not included during bootstrapping in the training dataset. Right? If so, I do not understand the reason for error difference.

The answer is that 25 patients have dependent measurements. But I do not get why it affected the test dataset and not out-of-bag error.

$\endgroup$
1

1 Answer 1

2
$\begingroup$

This is how the OOB error estimate is computed in RF: at any given step in the forest building process, any observation in the original data set can be fed to all the trees that were trained on bootsrap samples devoid of this observation. Approximately one third (~37%) of the total number of trees will meet this condition. Further, by letting these tree vote and taking the most popular class, a prediction can be obtained for the observation. The number of times the prediction differs from the true label of the observation averaged over all classes, gives the out-of-bag error estimate.

Breiman (2001) showed that this estimate was unbiased, and was very similar to Cross-Validation (CV), however, like CV, the way it estimates error is inherently limited by the training data. If specific scenarios or concepts are in an external testing set but absent from the training set (that is used for CV, or to compute OOB error estimate), this new signal won't be captured by the RF model. This will decrease its testing set performance, even though performance on the training set may seem very good as measured by OOB or CV.

$\endgroup$
4
  • $\begingroup$ so, it has nothing to do with the dependent measurements, right? $\endgroup$
    – Alina
    Jul 22, 2015 at 18:01
  • $\begingroup$ your not worried that the measurements are not independent? (and that the OOB estimate assumes independent observations) $\endgroup$
    – charles
    Jul 22, 2015 at 23:01
  • $\begingroup$ @Charles I don't think that the OOB estimate assumes anything. According to this paper by Breiman, algorithmic modeling (unlike parametric modeling) does not make any assumption about the underlying process from which the observations originate, nor about the true relationships between input and output variables. So there is no such thing as IID $\endgroup$
    – Antoine
    Jul 23, 2015 at 11:07
  • $\begingroup$ now, if some time pattern has to be captured about the signal, you need to state the problem more precisely: how was the training test constituted with respect to the time series? The testing set? $\endgroup$
    – Antoine
    Jul 23, 2015 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.