For a binary classifier, how do I evaluate the performance if I'm getting very high precision & recall values (~0.9) for one class, say A, but lower (~0.5-0.6) values for the other class, say B?
PS - The dataset is unbalanced, with greater no of observations for B but much less for A. Does that impact the model performance?
Unbalananced number of observations does impact f1 score when the number of observations for A and B are less, or the unbalance itself is too high. For example, 100000 samples for A and 150000 samples for B may not have such an impact, but 100 A samples and 150 B samples are likely to result in a classifier with low accuracy and misleading f1 score.
When I have reasonable accuracy for both classes and they are similar to each other, I will take their mean as the accuracy of the classifier. (f(A) + f(B))/2 where f(A) is the f1 score for class A. The other approach is to simply use a test sample set which includes equal samples for both classes A and B, and then computing total precision and recall.
If you, for some reason, can reasonably have training samples for only one class A and not for the other class B (in cases where B is anomaly observations, for example), then redesign your classifier as a single class classifier which classifies whether the test sample belongs to this known class or not.
