Random Forest: training error vs oob error

Question

When creating a random forest model, the training error is different from the out-of-bag error.

The out-of-bag error of the model is based on the predictions:

predict(model)

The training error of the model (rf) is based on the prediction:

predict(model,newdata=training_dataset)

Now my question is: if we create a random forest model, and saved the model. Then we loose the training_dataset... Then one day, we meet that training dataset again. How we can evaluate the training dataset?

Or we can consider a less dramatic situation: in the test dataset, there is an observation that is the same as we can find in the training dataset, how can we predict ? (if we know that it is from the training dataset, we would take the out-of-bag error, but now we consider that it is a new observation.)

More concretely, if obs_1=training_dataset[1,], we can calculate the prediction in two ways:

predict(model)[1]

or

predict(model,newdata=obs_1)

The result will be different. Which one should we consider ? For me, we should choose the first one (because there is a risk of overfitting in the second one).

And now, let's consider that in the this observation, only one variable changes value, let it be obs_1bis (let's say it is a numerical variable, the initial value is 1, and in the new observation, it is 1.001). Then the prediction will be very close to predict(model,newdata=obs_1) but it should be closer to predict(model)[1] if the previous consideration is correct.

EDIT:

If the oob error is, let's say, 10%. And the error based on predict(model,newdata=training_dataset) is 0%. Should we conclude that the model is heavily overfitted?

Untill now, I only look the oob error, and in the summary of the model of the R package, we only see this OOB estimate of error rate. Then using a test set data, the error rate would be not far from this error rate (10%).

Then I realized that if one observation is from the training dataset, and we consider newdata argument, then its prediction is different, hence my question.

Answer 1

Then one day, we meet that training dataset again. How we can evaluate the training dataset?

predict(model,newdata=training_dataset) will work in the usual way. Giving newdata to the model just makes a prediction for each row in newdata.

The object model stores the predictions for the OOB data, so losing the training data won't change the saved model.

Or we can consider a less dramatic situation: in the test dataset, there is an observation that is the same as we can find in the training dataset, how can we predict ?

The predictions happen in the usual way. Each tree in model makes a vote, and the votes are added up. The model doesn't care.

If you're very worried that having the same observations in train and test partitions will distort the model, then you can work out a stratification scheme to put all identical observations in either train or test.

The result will be different. Which one should we consider ?

It depends. OOB data is a way to simulate out-of-sample data using the training set. In the other hand, if you want a prediction that uses all of the trees (perhaps because you've deployed the model and need to apply it to new data), then you'd used predict(model, newdata=...).

For me, we should choose the first one (because there is a risk of overfitting in the second one).

Overfitting is a property of the model itself, not the mode you chose for predictions.

A prediction from OOB data might be overfit if the model is overfit. Or it might not be, if the model is not overfit. Either way, choosing predict(model, newdata=...) or predict(model) isn't a toggle that fixes overfitting.

And now, let's consider that in the this observation, only one variable changes value, let it be obs_1bis (let's say it is a numerical variable, the initial value is 1, and in the new observation, it is 1.001). Then the prediction will be very close to predict(model,newdata=obs_1) but it should be closer to predict(model)[1] if the previous consideration is correct.

We won't know ahead of time if a small change to 1 feature will cause a small or large change in the result, because trees are highly discontinuous. Decision trees find splits in the features -- if all of the trees split on this feature between 1.0 and 1.001, then the predictions could be very different. Or if the changed feature is never used, the predictions could be identical.

Answer 2

It is important that you understand how the OOB predictions are made.Each tree in the RF is trained on a bootstrap resample of the training data, and on average 1/3 of the original training data is not used in training each tree (not the same 1/3 for all trees).

For each training example, the OBB prediction for that data is the result of passing that data only to the trees that did not use it in the training, and compounding the result from each tree into a final decision (in RF the final decision is done by voting for classification and averaging for regression - I am 905 sure of the regression statement, and 100% sure of the classification statement).

The whole point of the OOB is to have an estimate of the error for unknown data. But that is only an estimate, since not all trees in the RF are contributing to the decision. It should be used only if you do not have a test set that is different than the training set. It could also be used to select the hyperparameters to the RF. But again this is only useful if either you do not have a separate training set, or you have very strict time issues - the OOB is calculates in the training, and thus you do not have to run a test phase - but testing is usually very fast in RF in comparison to training.

1. predict(model) is the way to retrieve the OOB results only from the R package RandomForest. Another frequently used random forest package in R is ranger which has a different interface.

Now you should realize that the OOB predictions are constructed during the training of the RF, and that it way it is an atribute of the model object.

2. predict(model)[1] is the prediction for the first OOB example (which I think is the first example in the training set) but as discussed above, using only the trees that did not include that example in their training.

predict(model,newdata=training.data)[1] is the prediction for the prediction for the first data in newdata, which in this case happens to be the training data. Which one should I consider? does not make much sense - they report very different things (none of which is much useful if you have a real separated test set).

3. 10% error in the OOB against 0% error in the training set seems to me an example of overfitting. 0% training error is already indicative of overfitting - your RF learned too much from the training set. Since OOB error is an estimate of the error for unseen data, you should expect something similar to 10% as test error.