I was working on a binary classification problem where the ratio of Y(Unprofitable)/N(Profitable) is 52/148, in a train set with a sample size of 200.
I got expected results with random forest model but I need help to interpret its result
Confusion Matrix and Statistics
Prediction N Y N 144 39 Y 6 11
Accuracy : 0.775 95% CI : (0.7108, 0.8309) No Information Rate : 0.75 **P-Value [Acc > NIR] : 0.2332** Kappa : 0.2308
Mcnemar's Test P-Value : 1.84e-06
Sensitivity : 0.2200 Specificity : 0.9600 Pos Pred Value : 0.6471 Neg Pred Value : 0.7869 Prevalence : 0.2500 Detection Rate : 0.0550
Detection Prevalence : 0.0850
Balanced Accuracy : 0.5900
'Positive' Class : Y
Here P Value[ACC>NIR] = 0.2308 which casts doubt on the fitness of the model but then what is Mcnemar's Test P-Value? I don't have any further validation/production data to test on. The business objective is to have a profitper Appln > base profit(i.e with no model).