2
$\begingroup$

Using Python's pandas, sklearn, and stats models API, I have trained a logistic regression on a training dataset that tells me whether or not a particular purchase (based on gender, and other features) was fraudulent. I held out a test set, and under a logistic regression all datapoints are predicted to be within %20 percent of unfraudulent (where fraudulent =0, unfradulent =1). In other words, arbitrarily setting a decision boundary to be 50%, all test points are predicted to be unfraudelent. This is because many many more data points are unfraudulent in the training set than fraudulent.

I am new to statistics, and would like to improve my model. What's the next step?

Improve this question
$\endgroup$
6
  • $\begingroup$ Unbalanced classes is tbe term to use, not sparse. Can you edit the title? And search this site for unbalanced classes $\endgroup$
    – kjetil b halvorsen ♦
    May 16, 2017 at 20:24
  • 1
    $\begingroup$ You are incorrectly evaluating your regression model. Instead if insisting on a hard classification threshold, you should evaluate your logistic regression on the basis of the probabilities it produces. Use the log-loss, not accuracy. $\endgroup$
    – Matthew Drury
    May 16, 2017 at 20:46
  • 1
    $\begingroup$ It sounds like your problem isn't unbalanced classes, but that your features don't predict fraud well! If your features don't help you forecast fraud, then your best prediction for every transaction is not fraudulent. If that's what's going on, the next step would be to build/get a richer feature set. $\endgroup$
    – Matthew Gunn
    May 16, 2017 at 21:17
  • $\begingroup$ If you're running logistic regression, you could try construct 2nd degree polynomial terms over all your features, probably with some regularization if you have a lot of features. Or use a random forest classifier. Those would explore whether a more flexible functional form on your current data might do better. You want to estimate $\theta$ so that $f(x; \theta)$ forecasts fraud for input $x$. The two basic approaches are to get a better function $f$ or get better data $x$. $\endgroup$
    – Matthew Gunn
    May 16, 2017 at 21:34
  • $\begingroup$ @MatthewGunn I will try what you said, BUT you were right about the features. I found other features I could add in and no longer faced this problem, learning something in the process. Thanks again! $\endgroup$
    – user45254
    May 16, 2017 at 21:51

2 Answers 2

Reset to default
1
$\begingroup$

  1. You can re-set the decision boundary by maximizing the F-score, which takes into consideration both precision and sensitivity. https://en.wikipedia.org/wiki/F1_score

  2. Without too much added information, you can also set the decision boundary as the probability of fraudulent in your training dataset. Any record with predicted probability above average will be identified as 'fraudulent'.

  3. Another way to decide the cut-off probability is by assigning 'loss' to each case of misclassification. For instance, non-fraudulent misidentified as fraudulent costs 1, while fraudulent misidentified as non-fraudulent costs 10. Then the optimal decision boundary will be the one that minimizes the sum of loss in your training data. It is more subjective this way, but you can customize your model with some personal beliefs.

Improve this answer
$\endgroup$
0
$\begingroup$

Did you split your datasets respecting the base rates? If you did not, you could introduce some bias, as logistic regression intercept reflects base rates. If that's the case, it is possible to simply correct the intercept term (the other parameters need not change). There is a paper describing this approach by King and Zeng , I think in 2001.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.