What measure of training error to report for Random Forests?

Question

I'm currently fitting random forests for a classification problem using the randomForest package in R, and am unsure about how to report training error for these models.

My training error is close to 0% when I compute it using predictions that I get with the command:

predict(model, data=X_train)

where X_train is the training data.

In an answer to a related question, I read that one should use the out-of-bag (OOB) training error as the training error metric for random forests. This quantity is computed from predictions obtained with the command:

predict(model)

In this case, the OOB training error is much closer to the mean 10-CV test error, which is 11%.

I am wondering:

1. Is it generally accepted to report OOB training error as the training error measure for random forests?

2. Is it true that the traditional measure of training error is artificially low?

3. If the traditional measure of training error is artificially low, then what two measures can I compare to check if the RF is overfitting?

Answer 1

To add to @Soren H. Welling's answer.

1. Is it generally accepted to report OOB training error as the training error measure for random forests?

No. OOB error on the trained model is not the same as training error. It can, however, serve as a measure of predictive accuracy.

2. Is it true that the traditional measure of training error is artificially low?

This is true if we are running a classification problem using default settings. The exact process is described in a forum post by Andy Liaw, who maintains the randomForest package in R, as follows:

For the most part, performance on training set is meaningless. (That's the case for most algorithms, but especially so for RF.) In the default (and recommended) setting, the trees are grown to the maximum size, which means that quite likely there's only one data point in most terminal nodes, and the prediction at the terminal nodes are determined by the majority class in the node, or the lone data point. Suppose that is the case all the time; i.e., in all trees all terminal nodes have only one data point. A particular data point would be "in-bag" in about 64% of the trees in the forest, and every one of those trees has the correct prediction for that data point. Even if all the trees where that data points are out-of-bag gave the wrong prediction, by majority vote of all trees, you still get the right answer in the end. Thus basically the perfect prediction on train set for RF is "by design".

To avoid this behavior, one can set nodesize > 1 (so that the trees are not grown to maximum size) and/or set sampsize < 0.5N (so that fewer than 50% of trees are likely to contain a given point $(x_i,y_i)$.

3. If the traditional measure of training error is artificially low, then what two measures can I compare to check if the RF is overfitting?

If we run RF with nodesize = 1 and sampsize > 0.5, then the training error of the RF will always be near 0. In this case, the only way to tell if the model is overfitting is to keep some data as an independent validation set. We can then compare the 10-CV test error (or the OOB test error) to the error on the independent validation set. If the 10-CV test error is much lower than the error on the independent validation set, then the model may be overfitting.

Answer 2

[edited 21.7.15 8:31 AM CEST]

I suppose you used RF for classification. Because in this case, the algorithm produces fully grown trees with pure terminal nodes of only one target class.

predict(model, data=X_train)

This line of coding is like a dog chasing [~66% of] its own tail. The prediction of any training sample is the class of the training sample itself. For regression RF stops if node has 5 or less samples in it or if node is pure. Here prediction error will be small but not 0%.

In machine learning we often work with large hypothesis spaces. This means there will always be many not yet falsified hypothesis/explanations/models to data structure of our training set. In classical statistics is the hypothesis space often small and therefore the direct model-fit is informative accordingly to some assumed probability theory. In machine learning does the direct lack-of-fit relate to the bias of the model. Bias is the "inflexibility" of the model. It does not in anyway provide a approximation of generalization power(the ability to predict new events). For algorithmic models cross-validation is the best tool to approximate generalization power, as no theory is formulated. However, if model assumptions of independent sampling fail, the model may be useless anyhow, even when a well performed cross-validation suggested otherwise. In the end, the strongest proof is to satisfyingly predict a number external test-sets of various origin.

Back to CV: Out-of-bag is often a accepted type of CV. I would personally hold that OOB-CV provides similar results as 5-fold-CV, but this is a very small nuisance. If to compare let's say RF to SVM, then OOB-CV is not usefull as we would normally avoid to bag SVM. Instead then both SVM and RF would be embedded in the exact same cross-validation scheme e.g. 10-fold 10-repeats with matching partitions for each repeat. Any feature engineering steps would often also be needed to be cross-validated. If to keep things clean the entire data pipe-line could be embedded in the CV.

If you tune your model with your test-set(or cross-validation) you're again inflating your hypothesis space and the validated prediction performance is likely over-optimistic. Instead you will need a calibration-set(or calibration CV-loop) to tune and a test validation set(or validation CV-loop) to assess your final optimal model.

In the extreme sense, your validation score will only be unbiased if your never act on this result, when you see it. This is the paradox of validation, as why would we obtain a knowledge which is only true if you do not act on it. In practice the community willingly accepts some publication bias, where those researchers who got a over-optimistic validation at random are more likely to publish, than those who unluckily good a over-pessimistic validation. Therefore sometimes why can't reproduce others models.