4
$\begingroup$

I'm currently training two separate Random Forest classifier models using a dataset where the target feature is imbalanced (fraud): RF 1 is trained on the imbalanced data and RF 2 is trained on SMOTE-applied data.

Both models are trained with n_estimators = 300 and make use of train, test and validation sets. (I will move to cross-validation later on in my analysis)

Here are the results:

Results of Random Forest fitted on imbalanced data:
 
Recall Training: 1.0
Recall Validation: 0.8485299590621511
Recall Test: 0.8408843783979703
-
Accuracy Training: 1.0
Accuracy Validation: 0.9398255813953489
Accuracy Test: 0.9343023255813954
-
F1 Training: 1.0
F1 Validation: 0.9167671893848011
F1 Test: 0.9112333071484682
-
OOB-Error 0.06497211068515052
 
###################################################################
 
Results of Random Forest fitted on SMOTE-applied data:
 
Recall Training: 1.0
Recall Validation: 0.8489021213248976
Recall Test: 0.8412468285610728
-
Accuracy Training: 1.0
Accuracy Validation: 0.9392441860465116
Accuracy Test: 0.9341569767441861
-
F1 Training: 1.0
F1 Validation: 0.9160642570281123
F1 Test: 0.911089303238469
-
OOB-Error 0.05621284006966554

As you can see, both RFs score virtually the same across all performance metrics apart from the OOB-Error in which the second RF scores better (I think?).

My question is how am I supposed to interpret the OOB-Error when looking that the other performance metrics. In the first RF, the OOB-Error is 0.064 - does this mean for the OOB samples, it predicted them with an error rate of 6%? Or is it saying it predicts OOB samples correctly 94% accuracy/confidence? I understand what OOB errors and samples are but I'm struggling to interpret this metric intuitively.

Thank you in advance.

$\endgroup$
3
$\begingroup$

Yes, out-of-bag error is an estimate of the error rate (which is 1 - accuracy) that this training approach has for new data from the same distribution. This estimate is based on the predictions that you get for each data point by using only averaging those trees, for which the record was not in the training data.

If you have a low number of trees the OOB error might be not a good estimate of the error rate that training an algorithm like this has on new data, because each tree in RF tends to be underfit and only once you combine enough trees the RF gets better (so if there's only a small number of trees per record, it may underestimate performance). On the other hand, once you have a huge number of trees it becomes a pretty good estimate like you get from a train-validation split with a lot of data (or cross-validation).

What makes it difficult to interpret in your particular case is that the OOB error is estimated on the training data distribution. When you do SMOTE, you change the training data distribution vs. the true distribution, which affects accuracy (which is of course a performance measure that is affected by class prevalence). So, I would not compare OOB error between the two scenarios you have for that reason.

$\endgroup$
3
  • $\begingroup$ hello, @bjorn - thank you for making this clearer. Could you elaborate on your final point: "...the OOB error is estimated on the training data distribution. When you do SMOTE, you change the training data distribution vs. the true distribution, which affects accuracy" Are you saying looking at OOB Error for SMOTE applied data isn't effective as it's not taking into consideration the true data distribution? $\endgroup$
    – softmax55
    Aug 20 at 9:57
  • 1
    $\begingroup$ E.g. let's say before SMOTE you had 1% Yes & 99% No. You used SMOTE to get 50:50. When you use bagging, you are randomly leaving out some of the training data for each tree & that left-out data is 50:50 Y:N. From the OOB error, you get performanmce one data generated using SMOTE with 50:50 Y:N, but not performance with the true data distribution incl 1:99 Y:N. Even if SMOTE generates data that's as good as if you collected more data, the class imbalance affects accuracy (e.g. super simple example: with 50:50 you get 50% accuracy by always predicting yes, with 1:99 that only gets 1% accuracy). $\endgroup$
    – Björn
    Aug 20 at 10:11
  • $\begingroup$ Okay, I get it now, thank you @Bjorn - have a good day $\endgroup$
    – softmax55
    Aug 20 at 10:21
0
$\begingroup$

The results suggest that the random forest that you are using only predict the OOB samples with 94% accuracy. As it is an error rate, you can think about it as the number of wrongly classified observations

There is a good writing about this topic on a blog where the author sums up the concept: https://towardsdatascience.com/what-is-out-of-bag-oob-score-in-random-forest-a7fa23d710

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.