I was working on a binary classification problem where the ratio of Y(Unprofitable)/N(Profitable) is 52/148, in a train set with a sample size of 200.

I got expected results with random forest model but I need help to interpret its result



Confusion Matrix and Statistics


Prediction N Y N 144 39 Y 6 11

          Accuracy : 0.775           
             95% CI : (0.7108, 0.8309)
No Information Rate : 0.75            
**P-Value [Acc > NIR] : 0.2332**          

              Kappa : 0.2308          

Mcnemar's Test P-Value : 1.84e-06

        Sensitivity : 0.2200          
        Specificity : 0.9600          
     Pos Pred Value : 0.6471          
     Neg Pred Value : 0.7869          
         Prevalence : 0.2500          
     Detection Rate : 0.0550          

Detection Prevalence : 0.0850
Balanced Accuracy : 0.5900

   'Positive' Class : Y                                      

ExpectedProfitPerAppl= ((4000*(144-6))-(4900*39))/200

ExpectedProfitPerAppl 1804.5

Here P Value[ACC>NIR] = 0.2308 which casts doubt on the fitness of the model but then what is Mcnemar's Test P-Value? I don't have any further validation/production data to test on. The business objective is to have a profitper Appln > base profit(i.e with no model).

  • 2
    $\begingroup$ you got it right: the forest is quite bad (it may be better than nothing though). at least try some other method $\endgroup$ – carlo Dec 28 '19 at 21:55
  • $\begingroup$ What does McNemar test have to do here? You are not comparing two models, so I am a bit confused on how you would like to use it... Can you expand? $\endgroup$ – Davide ND Feb 5 '20 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.