How to handle predictors that are highly correlated to the response

Question

First off, I'd like to say that i'm not very experienced with statistics and not a native English speaker so feel free to tell me if my question is completly obvious or unclear.

I'm currently working on a unbalanced dataset ( fraud detection) and I have a problem with a few variable categories that are highly correlated to the response. For example, there's a variable that represents payment type. Most people trying to fraud have chosen checks and a third of people who have chosen checks are frauds so my model tends to classify as a fraud any observation with check as the payment type.
The correlation between the two variables is ~50% and the precision i get after applying bagged Cart is around 0.4 and other algorithms also work rather poorly.

How can I deal with this without removing the variable from my dataset?

Answer 1

It sounds like what you have is a powerfully predictive variable, and there is no reason to remove it.

What you have to watch out in situations like this is what is called leakage. Leakage is when you have a predictor that is just some version of your response in disguise.

For example, suppose that you have a system at your company that, when fraud is detected, first switched the account into "investigation" status, and then when the investigation is complete, cancels it due to fraud. The "investigation status" will look like a very powerful variable, but it is caused by the response (fraud). If you went to implement your model, attempting to detect fraud, then the "investigation status" variable would be useless, as if an account is in investigation status, you already know its fraudulent.

You can see why this is called leakage, the response has "leaked into" the predictors.

So, think carefully about whether this could be the case with your account status, but I suspect not. In that case, you just have a really good predictive variable.

Most people trying to fraud have chosen checks and most people that have chosen checks and a third of people who have chosen checks are frauds so whenever my model tends to classify as a fraud any observation with check as the payment type.

You shouldn't evaluate your model by classifying records as fraud or non fraud. Instead, you should get your model to assign probabilities of fraud to each evaluation record, and work directly with those probabilities. If you use this context, then your issue here goes away, as you will simply observe that using a check gives a high probability of fraud, which does not mean that all check users are fraudulent.

Answer 2

In addition to @Matthew Drury's answer, you could train different models for different transaction methods (cheques, non-cheques). This way the features of people using cheques would be highlighted and the column will also remain in the data. See if the tool you are using allows grouping when fitting models. This could save additional work.

Answer 3

Imbalance

By far the cleanest* way to (50%-50%) balance your training set is to use observation weights. Say 1% of cases is fraudulent, 99% is not, then give each non-fraudulent case weight 1, and each fraudulent transaction weight 99. This is equivalent to adding 98 copies of each fraudulent cases back into the training set.

For example, in R's rpart or lm use argument weights in randomForest set classwt = c(0.5, 0.5). In Python's sklearn.tree.DecisionTreeClassifier set class_weight = "balanced".

Doing so will make the classifier "work harder" on the minority outcome class (fraud), as misclassifying it now carries a highly increased cost.

*I write "cleanest" because (randomly) oversampling creates unnecessary sampling noise as certain cases are sampled more or less than 99 times. Weighting, instead, treats all cases equally.

Decision Trees

Given that you are using a classification tree, you could force the node check == yes to split, so that you do not have to classify all cases that end up in that node as fraudulent (of course I'm speculating -- I don't know your data).

Lastly, classification trees naturally output probabilities. Say a new case falls into an end-node (a leaf) that consists of 72% fraudulent cases and 28% non-frauds, then the estimated probability that the new case is fraudulent equals 72%.