# Visualizing the logistic regression cut-off threshold

I have a binary dependent variable dataset to be inputted into a pre-trained logistic regression model.

I need to choose an appropriate cutoff threshold $P$ such that for a data point $x_i$ with output probability $p_i$, if $|0.5 - p_i| < P$ then the classification of $x_i$ is considered unreliable and a null-call is returned.

Generally there a several competing metrics for determining the appropriate threshold $P$, the main three I came up with are:

1. The accuracy of the null-call data points, i.e. those $x_i$ such that $|0.5 - p_i| < P$.

2. The accuracy of the kept data points, i.e. those $x_i$ such that $|0.5 - p_i| \geq P$.

3. The proportion of data points which will be null-called.

Obviously I would like metrics 1 and 3 to be as small as possible, and metric 2 to be as large as possible.

After some thought, here is the best visualization I could come up with:

As you can see, the x-axis is transformed to metric 3, rather than just the threshold $P$, that way you can get a to scale sense of how much of a penalty you pay in null-calls by shifting farther to the right.

In general you're looking for x-values where there is a large gap between the green and red lines, and thus the naive choice would be the degenerate threshold $x=0$, however since a binary classifier is only useful when it does better than random choice, there's no downside to choosing a null-accuracy as high as $0.5$, and since you get a kept-accuracy boost at the null-accuracy $0.5$ trough (at approximately $x=0.05$), this is clearly the optimal choice.

The one thing I don't like about this plot is that the scales of the accuracies both seem poorly chosen, and for different reasons (which makes it extra difficult to find a y-axis scale which would work for both).

For the null-accuracy line, there's no reason to ever keep a region of data points where the average classification accuracy in that region is 50% or less. Unfortunately the graph makes it look like there's a significant difference between a null-accuracy of 0% and 50%, when in reality there's no important difference.

For the kept-accuracy line, the graph makes it look like the jump in kept-accuracy at around $x=0.05$ is small and thus no big deal, when in reality it is a big deal.

Anyways, I'm curious if anyone has any suggestions for improving this visualization, or even completely different alternative visualizations. I was thinking maybe some variation of an ROC curve might work, but I couldn't come up with a variation which I felt worked as well as the above visualization.