Accuracy of training sample in Random Forest model in R

Question

I'm using a Random Forest algorithm in order to construct a classification model, and I HAVE to check the accuracy of my rf model in the training sample, but as you can see in this answers :

https://stats.stackexchange.com/a/112052/90446

https://stats.stackexchange.com/a/66546/90446

you can't evaluate the accuracy considering the training samples like this:

predict(model, data=train)

I'm not confortable with the idea of use OOB to get accuracy of the training sample, because the OOB was not used to build the model, how could this be right? I don't know what should I do to get the accuracy of the training sample, is it possible or make any sense? When a check the AUC of the prediction of my training sample I get something near of 0.98, but the AUC of the test sample is about 0.7. Is this due to the limitations of prediction at the training sample or due to Overfitting?

Answer 1

In machine learning, model error on the training data is meaningless, period. It is not a limitation nor overfitting, merely a consequence of the fact that methods are built just to perform well on unseen data.

In case of RF "train set error" is expected to be near zero because RF uses not-pruned decision trees, hence naturally looks suspicious; yet in case of other algorithms you can expect strange biases and unexpected behaviours as well.

OOB is neither some kind of "training set error which looks good", but a result of an internal cross-validation; it should be also rather used as a diagnostic, and, if compared, compared with results of cross-validation of other approaches.

Answer 2

I agree on splitting the data into test and train, with that

predict(model, data=train)->Prediction

From the caret package,

postResample(Prediction, test$y)

this function will give you Accuracy and Kappa.

Other way:

mean(Prediction == test$y)