2
$\begingroup$

I am developing a fraud detection model with unbalanced data set , I used Random Forests Algorithm.

The confusion matrix showed that model has good sensitivity(recall) but low precision(Pos Pred Value). my question is

How to deal with low precision confusion matrix? do I need to try different Algorithm or try different tuning parameters?

$\endgroup$
2
+50
$\begingroup$

I'd say you don't need to change the algorithm you're approaching this problem with, but definitely your approach to working with your resulting model. In applications like fraud predicition it is quite common to have quite imbalanced class distributions, as it is often the case that there are for example way fewer fraudulent transactions submittetd to a credit card company than legitimate ones.
This is one part of it, but another is: what consequence does your models' decision have? Both scenarios of a wrong classification are bad - misclassifying a fraudulent transaction as legit, as well as misclassifying a legit one as fraudulent. But how bad are these? This depends highly on the context of the analysis, so a general answer is difficult to propose, but there are costs associated with both.

Your question states that your model shows low precision, so the amount of truly fraudulent transactions among those that are classified as fraudulent is low, that is a large number of your "positive" predictions are wrong. What does that mean for your application? This could for example mean that your company needs to allocate an employee to verify that transaction, incurring a certain workload as well as possible dissatisfaction of a customer that you are bothering with this.

The question you should be investigating is: how certain does the classification of my model need to be to justify the potential cost of a missclassification?
A random forest can provide you with a probability estimate for a given class belonging to either class, which is a good starting point. As Stephan Kolassa pointed out in this answer your real problem is about how to make a decision given the estimates of your model.
There are different ways you can go about this, off the top of my head I would suggest evaluating different probability thresholds for your classification with scoring rules, as mentioned in the linked answer. Classifying an instance as positive because the associated probability estimate is simply higher than that for negative doesn't necessarily make for a good basis for decision - maybe it is best to only classify instances as positive if $P(y=+ | X) >0.9$ ?
Another approach to evaluate probability thresholds would be by defining a cost matrix in which you specify a numeric cost for misclassifying positives and negatives respectively, then generate a confusion matrix based on a probability threshold for your classification and then see which threshold gives you the smallest error.

Either way, finding a fitting probability threshold for your problem should help making right decisions based on your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.