0
$\begingroup$

I am trying to compare performances of different classifiers to predict some data. The variable I wand to predict is binary (A or B), and I have a bunch of predictive variables to predict to what class each point belongs. The final utility of these classifiers will be to predict the number of individuals belonging to group A and B in different treatments, like in the picture below. I will particularly be interested in comparing the proportion of individuals belonging to A or B between different treatments.

Proportion of each class in each treatment

I am wondering what metrics I should use to compare these models. What I want to do is to have a classifier that will predict as well as possible the proportion of individuals belonging to class A and B in a given population. Intuitively, the ability to do so will depend on accuracy but also on false positive rates and false negative rates.

Is there a metrics that evaluates a classifier's ability to predict the proportion of the two classes in different populations (may they be balanced or unbalanced) ? I was thinking about MCC but I'm not sure I understood it correctly.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Answering the first question, i think some errors measures using cross validated techniques such as leave-one-out will tell you wich classifier had the best result

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.