Random Forest, dependent measurements

Question

I have a following quiz:

A random forest is used for classifying the disease state of patients based on measuring multiple genes. The dataset consists of 100 genes and 50 patients. However, for each patient, 10 measurements exist at different time points.

When analyzing the data, the out-of-bag error is very close to the error on the training data, but very different from the error on the test dataset.

I am not quite sure if I understand it correctly. The out-of-bag error is calculated on the points (patients) that were not included during bootstrapping in the training dataset. Right? If so, I do not understand the reason for error difference.

The answer is that 25 patients have dependent measurements. But I do not get why it affected the test dataset and not out-of-bag error.

Answer 1

This is how the OOB error estimate is computed in RF: at any given step in the forest building process, any observation in the original data set can be fed to all the trees that were trained on bootsrap samples devoid of this observation. Approximately one third (~37%) of the total number of trees will meet this condition. Further, by letting these tree vote and taking the most popular class, a prediction can be obtained for the observation. The number of times the prediction differs from the true label of the observation averaged over all classes, gives the out-of-bag error estimate.

Breiman (2001) showed that this estimate was unbiased, and was very similar to Cross-Validation (CV), however, like CV, the way it estimates error is inherently limited by the training data. If specific scenarios or concepts are in an external testing set but absent from the training set (that is used for CV, or to compute OOB error estimate), this new signal won't be captured by the RF model. This will decrease its testing set performance, even though performance on the training set may seem very good as measured by OOB or CV.