I have a binary dependent variable dataset to be inputted into a pre-trained logistic regression model.
I need to choose an appropriate cutoff threshold $P$ such that for a data point $x_i$ with output probability $p_i$, if $|0.5 - p_i| < P$ then the classification of $x_i$ is considered unreliable and a null-call is returned.
Generally there a several competing metrics for determining the appropriate threshold $P$, the main three I came up with are:
The accuracy of the null-call data points, i.e. those $x_i$ such that $|0.5 - p_i| < P$.
The accuracy of the kept data points, i.e. those $x_i$ such that $|0.5 - p_i| \geq P$.
The proportion of data points which will be null-called.
Obviously I would like metrics 1 and 3 to be as small as possible, and metric 2 to be as large as possible.
After some thought, here is the best visualization I could come up with:
As you can see, the x-axis is transformed to metric 3, rather than just the threshold $P$, that way you can get a to scale sense of how much of a penalty you pay in null-calls by shifting farther to the right.
In general you're looking for x-values where there is a large gap between the green and red lines, and thus the naive choice would be the degenerate threshold $x=0$, however since a binary classifier is only useful when it does better than random choice, there's no downside to choosing a null-accuracy as high as $0.5$, and since you get a kept-accuracy boost at the null-accuracy $0.5$ trough (at approximately $x=0.05$), this is clearly the optimal choice.
The one thing I don't like about this plot is that the scales of the accuracies both seem poorly chosen, and for different reasons (which makes it extra difficult to find a y-axis scale which would work for both).
For the null-accuracy line, there's no reason to ever keep a region of data points where the average classification accuracy in that region is 50% or less. Unfortunately the graph makes it look like there's a significant difference between a null-accuracy of 0% and 50%, when in reality there's no important difference.
For the kept-accuracy line, the graph makes it look like the jump in kept-accuracy at around $x=0.05$ is small and thus no big deal, when in reality it is a big deal.
Anyways, I'm curious if anyone has any suggestions for improving this visualization, or even completely different alternative visualizations. I was thinking maybe some variation of an ROC curve might work, but I couldn't come up with a variation which I felt worked as well as the above visualization.