When creating a random forest model, the training error is different from the out-of-bag error.
The out-of-bag error of the model is based on the predictions:
The training error of the model (rf) is based on the prediction:
Now my question is: if we create a random forest model, and saved the model. Then we loose the training_dataset... Then one day, we meet that training dataset again. How we can evaluate the training dataset?
Or we can consider a less dramatic situation: in the test dataset, there is an observation that is the same as we can find in the training dataset, how can we predict ? (if we know that it is from the training dataset, we would take the out-of-bag error, but now we consider that it is a new observation.)
More concretely, if
obs_1=training_dataset[1,], we can calculate the prediction in two ways:
The result will be different. Which one should we consider ? For me, we should choose the first one (because there is a risk of overfitting in the second one).
And now, let's consider that in the this observation, only one variable changes value, let it be
obs_1bis (let's say it is a numerical variable, the initial value is 1, and in the new observation, it is 1.001).
Then the prediction will be very close to
predict(model,newdata=obs_1) but it should be closer to
predict(model) if the previous consideration is correct.
If the oob error is, let's say, 10%. And the error based on
predict(model,newdata=training_dataset) is 0%. Should we conclude that the model is heavily overfitted?
Untill now, I only look the oob error, and in the summary of the model of the R package, we only see this OOB estimate of error rate. Then using a test set data, the error rate would be not far from this error rate (10%).
Then I realized that if one observation is from the training dataset, and we consider
newdata argument, then its prediction is different, hence my question.