0
$\begingroup$

I have a question regarding practical application of the probabilities output by predict_proba. I have a binary classification set, which I've trained a ExtraTreesClassifier on. For the problem I get the following metrics:

Classification Report:

                 precision    recall  f1-score   support

          0       0.79      0.69      0.73      1388
          1       0.59      0.72      0.65       887

avg / total       0.71      0.70      0.70      2275

I get a confusion matrix like this:

[[952 436]
 [251 636]]

Mathews Correlation Coefficient is 0.39366467656087517

With a Swarmplot of the data plotting actual class vs probability of being class 1, it looks like this:

Swarmplot

In my application I care primarily about true positives and false positives. I don't care about false negatives. If I choose a threshold for class 1 that is above 0.5 probability I get a higher proportion of true positives to false positives. At a particular threshold the True and False Negatives reach 0, at which point the MCC will also correctly be 0, indicating that the model is no better than random guessing.

I'm also likely to at some point reach an optimal level of True positive vs False positive, but at that point it's also likely MCC is 0 because there are no negative class items there.

Is this a valid application of the use of the probability field? Matthews Correlation Coeffient seems to indicate that it may not be, but I'm not sure that I'm properly applying it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.