2
$\begingroup$

I'm modeling credit fraud, where I have a small number of samples that result in fraud (1), and most samples that are not fraud (0). I am creating a models for detecting fraud based on new data.

I'm using the following models: logistic regression, K-nn, Support Vector Classifier and decision tree. The dataset is very similar to this: kaggle.com/mlg-ulb/creditcardfraud

I performed random undersampling on the data to get a 1:1 ratio. This made my models perform a lot better, but since the undersampling is performed randomly every time, I get a slightly different result because of the chosen samples.

Is there a way to find out which of the 8200 majority class samples are best to use in the undersampled data?

I was hoping to figure out which these are and only use them with the 800 positive samples on my undersampled dataset.

Improve this question
$\endgroup$
8
  • $\begingroup$ Could you tell us some context, what does your data represent, and most importantly, what is your goal? Without answers to this we are let to guess, and would have to close the question! But, undersampling is probably not the solution, better methods is! See stats.stackexchange.com/questions/247871/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jan 25, 2020 at 14:40
  • $\begingroup$ I'm modeling credit fraud, where I have a small number of samples that result in fraud (1), and most samples that are not fraud (0). I am creating a models for detecting fraud based on new data. The dataset is very similar to this: kaggle.com/mlg-ulb/creditcardfraud Even though undersampling might not be the optimal soltuion, like you said, I'm still trying to explore the results it produces. $\endgroup$
    – ire
    Commented Jan 25, 2020 at 14:46
  • $\begingroup$ What kind of model did you use, logistic regression or what? And, please, add new information as an edit to your original post. Noe everybody reads comments .. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jan 25, 2020 at 14:48
  • 1
    $\begingroup$ Those all depend on making a classification, that is, a hard decision. For logistic regression, that depends on defining a probability threshold for classification. How did you do that? undersamplig effectively changes the probability threshold, but you could do that directly, with the complete data set. Would probably have a similar effect. But you should better view your problem as risk estimation, not classification, and use some proper scoring rule. Read carefully stats.stackexchange.com/questions/127042/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jan 25, 2020 at 16:45
  • 1
    $\begingroup$ Thank you for your help. $\endgroup$
    – ire
    Commented Jan 25, 2020 at 17:18

3 Answers 3

Reset to default
2
$\begingroup$

The methods you list in the question are discriminative (i.e. you do not indicate that you used the one-class variety e.g. of SVM). Have you considered one-class classification?

One class classification is probably better suited for credit fraud detection than discriminative classification and as a side effect it does (by construction) not have any difficulties with class imbalance.

One-class classification is good for situatione where

  • a* well-defined ("positive") class is to be separated from cases that do not belong to this class.
    • well-defined means that the data points of this class form a (or maybe few separate) clouds, but we need not expect "surprises" in future cases (this is really not different from how we think of classes in discriminative settings)
    • and there are sufficient cases available from this class
    • * there can be several such classes, they are treated separately.

  • cases that do not belong to the positive class (aka the negative class) are ill-defined:

    • We cannot (or need not be able to) describe those cases better than saying they do not belong to the positive class.
    • There may be many reasons (classes) of cases that do not belong to the positive class
    • any new case that does not belong to the positive class may not belong there for an entirely new reason.
    • Side note: specific known classes within the negatives can be modeled as their own class - this does not affect the performance of the recognition of the positive class in any way.

  • In one-class classification, a case can belong to more than one class (consider medical diagnosis: a patient may have several diseases that are independent of each other).

Your task sounds to me as if the no-fraud cases are a prime example for a positive class. In addition, if you have examples of specific known types of fraud, you can also model them as their own class. If only few cases are available, recognition of that that (sub)class of fraud will of course be uncertain. But this will not disturb the performance of recognizing no-fraud cases.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ Thanks for the recommendation. I have now tried a One-Class classification model (using One-Class Adversarial Nets) on my data and though the results are slightly worse than those with my other models I really like the concept. $\endgroup$
    – ire
    Commented Jan 29, 2020 at 9:59
  • $\begingroup$ @ire: one-class classifiers usually require larger sample size than discriminative classifiers to reach the same certainty in the class boundaries. I see this as similar to the relationship non-parametric estimates vs. parametric estimates: stronger assumptions allow conclusion with less data (for a discriminative classifier, the other class helps fixing the boundary). Of course only if the assumptions are met. In that sense, discriminative models sometimes only look better at first glance. Does your better/worse include tests with fraud types that were not in the training? $\endgroup$
    – cbeleites
    Commented Jan 29, 2020 at 21:54
  • $\begingroup$ (assuming that in your task new, unknown types of fraud may occur) $\endgroup$
    – cbeleites
    Commented Jan 29, 2020 at 21:54
  • $\begingroup$ For my logistic regression model I did include tests with data that was not in the training. I'm going to do the same for the Adversarial Nets model and also try One class SVM. Like you said though, my dataset is probably not big enough for the latter models. $\endgroup$
    – ire
    Commented Jan 30, 2020 at 8:48
  • 1
    $\begingroup$ @ire: I was refering not only to testing with unknown (new) cases but cases where a new type of fraud occurs. I don't know sufficient about credit fraud to suggest an example, but here's a hypothetical example from my field: task is to recognize good/bad production batches. Good is the positive class, but various things can go wrong and cause the batch to be bad. Assume you had a couple of bad batches available for training (let's assume temperature and/or pressure were off, or the process was running not long enough or too long). Now a new bad batch comes in for testing. But the underlying $\endgroup$
    – cbeleites
    Commented Jan 30, 2020 at 11:49
1
$\begingroup$

You say (but in comments, it should be in the Q proper!) that you evaluated the models by precision, recall and f1 score. Those measure all depends on you making a hard decision, which in the case of say logistic regression needs a probability threshold. You didn't say how you did choose that threshold, so assuming you did use the "default" of one half. There is in general no convincing reason to always use that default.

Now, using undersampling of the majority class, you effectively changed the threshold (in terms of the complete data.) If you just used a different threshold with the complete data, you probably would have seen similar results. It would be better for you to see the problem as one of risk estimation, and then evaluate the models using some proper scoring rule. Then you do not need to make a hard decision. Please read carefully Why isn't Logistic Regression called Logistic Classification? (including its links and references.)

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I'm using python's sklearn LogisticRegression where the probability threshold is 0. Thank you for pointing me in a direction I wouldn't otherwise consider. $\endgroup$
    – ire
    Commented Jan 27, 2020 at 14:33
0
$\begingroup$

I have found some undersampling algorithms which can be used to find out which of the majority class samples are closest to the minority ones.

Some of these algorithms are Near Miss Undersampling, Condensed Nearest Neighbor Rule for Undersampling,...

Using near miss data in undersampling causes massive overfitting in my model so this approach isn't viable though.

Source with reproducible examples: https://machinelearningmastery.com/undersampling-algorithms-for-imbalanced-classification/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.