Classification, create a grey zone

I want to create a "grey zone" for a binary classifier. Grey zone means, in this zone classifier should give the result "I don't know".

I denote classes with + and -.

I have good tpr , but not so good fpr. Hence I want to do

1. Change thresholds separately for class positive and negative. For instance:

If instances has probability of belonging to class + higher than 0.9 -> +

If instances has probability of belonging to class - higher than 0.7-> -

Rest is grey zone, for instance 0.5 probability for class +, 0.5 probability for class - -> grey zone

How should I set up thresholds for two classes, would ROC threshold help?

• Start from the beginning. How many classes do you have? What are their names? Does every example belong to one of their classes? May 22 '16 at 0:38
• Of possible interest: stats.stackexchange.com/a/469059/247274. Kolassa's discussion about a "[SUSPECTED SPAM]" in the subject line is particularly pertinent.
– Dave
Sep 29 '20 at 21:37

Simple method: Plot the ROC curve, tip it to the right and mark the highest point. Rotate it further and mark the highest point again. The examples corresponding to those points are at your thresholds. You can tune your results by changing how far you tilt the graph.

Rigorous method with explanation: One good way to turn an ROC curve into a threshold is with a cost function - assign a cost per false positive and a different cost per false negative, evaluate the total cost of each point on your curve and choose the point with the lowest cost. In your case need to pick two thresholds, one between + and ? and another between ? and -. Use different cost functions, giving the +? boundary a higher cost for false positives and lower cost for false negatives than the ?- boundary. And ideally you'd derive the costs from how the model is actually used - how bad is a "maybe" compared to a genuine mistake, and how bad is a "maybe" instead of a correct answer?

The two methods give the same results if the slope of the tilt is the ratio of your false negative cost to false positive cost.

I can think of two alternatives (there might be others too) :

1. Learn a regression instead of classification. This way you obtain a "positiveness" score, and you can decide based on its value and your thresholds (you can also optimize them). Logistic regression might be helpful for this.
2. If you already know which samples to say "I don't know", make them another class, so learn a 3-class classification.