1
$\begingroup$

I have a binary dependent variable dataset to be inputted into a pre-trained logistic regression model.

I need to choose an appropriate cutoff threshold $P$ such that for a data point $x_i$ with output probability $p_i$, if $|0.5 - p_i| < P$ then the classification of $x_i$ is considered unreliable and a null-call is returned.

Generally there a several competing metrics for determining the appropriate threshold $P$, the main three I came up with are:

  1. The accuracy of the null-call data points, i.e. those $x_i$ such that $|0.5 - p_i| < P$.

  2. The accuracy of the kept data points, i.e. those $x_i$ such that $|0.5 - p_i| \geq P$.

  3. The proportion of data points which will be null-called.

Obviously I would like metrics 1 and 3 to be as small as possible, and metric 2 to be as large as possible.

After some thought, here is the best visualization I could come up with:

enter image description here

As you can see, the x-axis is transformed to metric 3, rather than just the threshold $P$, that way you can get a to scale sense of how much of a penalty you pay in null-calls by shifting farther to the right.

In general you're looking for x-values where there is a large gap between the green and red lines, and thus the naive choice would be the degenerate threshold $x=0$, however since a binary classifier is only useful when it does better than random choice, there's no downside to choosing a null-accuracy as high as $0.5$, and since you get a kept-accuracy boost at the null-accuracy $0.5$ trough (at approximately $x=0.05$), this is clearly the optimal choice.

The one thing I don't like about this plot is that the scales of the accuracies both seem poorly chosen, and for different reasons (which makes it extra difficult to find a y-axis scale which would work for both).

For the null-accuracy line, there's no reason to ever keep a region of data points where the average classification accuracy in that region is 50% or less. Unfortunately the graph makes it look like there's a significant difference between a null-accuracy of 0% and 50%, when in reality there's no important difference.

For the kept-accuracy line, the graph makes it look like the jump in kept-accuracy at around $x=0.05$ is small and thus no big deal, when in reality it is a big deal.

Anyways, I'm curious if anyone has any suggestions for improving this visualization, or even completely different alternative visualizations. I was thinking maybe some variation of an ROC curve might work, but I couldn't come up with a variation which I felt worked as well as the above visualization.

$\endgroup$
0
$\begingroup$

Have a look at our r package / python module modelplot (modelplot.github.com). This helps you create 4 plots to decide on a proper cut-off value.enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.