I have a question regarding practical application of the probabilities output by predict_proba. I have a binary classification set, which I've trained a ExtraTreesClassifier on. For the problem I get the following metrics:
precision recall f1-score support 0 0.79 0.69 0.73 1388 1 0.59 0.72 0.65 887 avg / total 0.71 0.70 0.70 2275
I get a confusion matrix like this:
[[952 436] [251 636]]
Mathews Correlation Coefficient is 0.39366467656087517
With a Swarmplot of the data plotting actual class vs probability of being class 1, it looks like this:
In my application I care primarily about true positives and false positives. I don't care about false negatives. If I choose a threshold for class 1 that is above 0.5 probability I get a higher proportion of true positives to false positives. At a particular threshold the True and False Negatives reach 0, at which point the MCC will also correctly be 0, indicating that the model is no better than random guessing.
I'm also likely to at some point reach an optimal level of True positive vs False positive, but at that point it's also likely MCC is 0 because there are no negative class items there.
Is this a valid application of the use of the probability field? Matthews Correlation Coeffient seems to indicate that it may not be, but I'm not sure that I'm properly applying it.