How to calculate the best threshold value I'm working at a motor insurance company and want to build a business rule to detect fraud cases based on the damage value. I have a historical data set that contains a list of accidents info, damage values, and fraud status (fraud or not).
My goal is to identify the best threshold of the damage value so my business rule can detect fraud cases with an acceptable FP (fraud cases damage value tend to be high).
Any Idea on how to calculate the best threshold?
 A: This is entirely domain specific, there is no single "best threshold" that will apply to all possible scenarios. You can usually move a classification threshold to have higher sensitivity at the cost of lower specificity and vice versa, so understanding that tradeoff in your particular classifier is a good place to start. Where you ultimately want to draw the threshold depends very much on your particular use case and relative "cost" of false positives vs. false negatives. It sounds like the cost of false negatives is high (not detecting real fraud is a big problem for customers) but that false positives are not so costly (reviewing a no-fraud case just costs the company some time but does not impact customers).
You will need to quantify the relative cost of these different misclassifications and combine it with the sensitivity/specificity characteristics of your classifier to identify a threshold that has the highest overall utility with respect to misclassification cost. For example, if a false negative is twice as costly as a false positive, you may set the threshold one way, but if it's 1000 times as costly, you'd be better served by lowering the threshold and sending more cases for fraud review. In the limit where false negatives are catastrophic, you can't afford to miss any real fraud cases, and are forced to lower the threshold all the way and send everything to fraud review.
A: If you have a dataset containing labels you should build a supervised machine learning (ML) model, so that you could generalise information in the dataset and make predictions about new incidents.
If you are able to get incidents info for each incident, you can exploit this to train a ML model.
You could start with a simple baseline model like a Linear Discriminant Analysis, or even more simply a Naive Bayes using a binary-categorical distribution for the Likelihood.
Then you can increase the complexity of the model if the accuracy is too low for your use case.
