I am trying to compare performances of different classifiers to predict some data. The variable I wand to predict is binary (A or B), and I have a bunch of predictive variables to predict to what class each point belongs. The final utility of these classifiers will be to predict the number of individuals belonging to group A and B in different treatments, like in the picture below. I will particularly be interested in comparing the proportion of individuals belonging to A or B between different treatments.
I am wondering what metrics I should use to compare these models. What I want to do is to have a classifier that will predict as well as possible the proportion of individuals belonging to class A and B in a given population. Intuitively, the ability to do so will depend on accuracy but also on false positive rates and false negative rates.
Is there a metrics that evaluates a classifier's ability to predict the proportion of the two classes in different populations (may they be balanced or unbalanced) ? I was thinking about MCC but I'm not sure I understood it correctly.
