1
$\begingroup$

I'm using a Random Forest algorithm in order to construct a classification model, and I HAVE to check the accuracy of my rf model in the training sample, but as you can see in this answers :

https://stats.stackexchange.com/a/112052/90446

https://stats.stackexchange.com/a/66546/90446

you can't evaluate the accuracy considering the training samples like this:

predict(model, data=train)

I'm not confortable with the idea of use OOB to get accuracy of the training sample, because the OOB was not used to build the model, how could this be right? I don't know what should I do to get the accuracy of the training sample, is it possible or make any sense? When a check the AUC of the prediction of my training sample I get something near of 0.98, but the AUC of the test sample is about 0.7. Is this due to the limitations of prediction at the training sample or due to Overfitting?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

In machine learning, model error on the training data is meaningless, period. It is not a limitation nor overfitting, merely a consequence of the fact that methods are built just to perform well on unseen data.

In case of RF "train set error" is expected to be near zero because RF uses not-pruned decision trees, hence naturally looks suspicious; yet in case of other algorithms you can expect strange biases and unexpected behaviours as well.

OOB is neither some kind of "training set error which looks good", but a result of an internal cross-validation; it should be also rather used as a diagnostic, and, if compared, compared with results of cross-validation of other approaches.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. I understand that the evaluation metrics of error in the training sample are very biased because in RF there is no pruning, however I wouldn't expect that the training was near zero, because of the fact that the training sample also has the OOB with it(1/3 of the training sample), so this OOB is an unseen data and should increase the error. Because of this I'm wondering that this near zero error is due to overfitting or something else. $\endgroup$
    – Michael Elma
    Jan 9, 2016 at 16:49
0
$\begingroup$

I agree on splitting the data into test and train, with that

predict(model, data=train)->Prediction

From the caret package,

postResample(Prediction, test$y)

this function will give you Accuracy and Kappa.

Other way:

mean(Prediction == test$y)
Improve this answer
$\endgroup$
1
  • $\begingroup$ Welcome to our site! Note that you can get code formatting by selecting it and pressing control-K, or by using the button for code formatting in the editing window, or just starting the line with a bunch of spaces... I've done that for you, so you can press the "edit" button to see how I did it. Why not take a look at stats.stackexchange.com/editing-help so you can get your answers looking how you'd like them in future? $\endgroup$
    – Silverfish
    Mar 29, 2016 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.