12
$\begingroup$

I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1.

I've tried several classification algorithms, without too much fine tuning, just to get a feeling of how the baseline performance was. They all got an accuracy score of around 99%, that is exactly the ratio between class 0 samples and total samples. Artificially under-sampling just got the accuracy score down to the very same ratio of the new dataset, so no improvement on that side.

The F1 score is really bad because I'm experiencing awful Type II errors: basically, the algorithm is just guessing that everything is belonging to class 0. With some models that I tried, it literally predicts everything to be class 0: false positives are 0 (because no positive samples get predicted) and false negatives are really a lot (because actual positives are predicted to be negative). The AUC-ROC is around 50% (awful), and weighting the models to take into account the skewness of the classes brought no improvement.

I tried to do some feature engineering (ensembling the supervised classification on top of some unsupervised clustering), with almost no luck.

Do you have any suggestions regarding how to tackle such problems / how to diagnose the underlying issue(s) that prevent(s) the predictor from being accurate? Or I should take this as a proof that, given my the dataset, belonging to class 1 is just random (so I should collect more features)?

Side note: I thought about taking it from another side, ie. anomaly detection, but I'm not sure this could be the right approach.

$\endgroup$

2 Answers 2

12
$\begingroup$

Most of the classification problems I've tackled are similar in nature, so a large class imbalance is quite common.

It is not clear whether you are using training-validation sets to build and fine tune the model. Cross-fold validation is generally preferred since it gives more reliable model performance estimates.

The F1 score is a good classification performance measure, I find it more important than the AUC-ROC metric. Its best to use a performance measure which matches the real-world problem you're trying to solve.

Without having access to the dataset, I'm unable to give exact pointers; so I'm suggesting a few directions to approach this problem and help improve the F1 score:

  1. Use better features, sometimes a domain expert (specific to the problem you're trying to solve) can give relevant pointers that can result in significant improvements.

  2. Use a better classification algorithm and better hyper-parameters.

  3. Over-sample the minority class, and/or under-sample the majority class to reduce the class imbalance.

  4. Use higher weights for the minority class, although I've found over-under sampling to be more effective than using weights.

  5. Choose an optimal cutoff value to convert the continuous valued class probabilities output by your algorithm into a class label. This is as important as a good AUC metric but is overlooked quite often. A word of caution though: the choice of the cutoff should be guided by the users by evaluating the relevant trade-offs.

$\endgroup$
3
  • 6
    $\begingroup$ It is important that your learning algorithm predict class probabilities, and that a threshold is set for hard classification based on the risks of false negatives and false positives. I do not think your answer is complete without mentioning this. $\endgroup$ Apr 20, 2017 at 17:27
  • $\begingroup$ Do you usually fine tune threshold for F1 score after training a model is completed? $\endgroup$ Oct 14, 2020 at 9:02
  • $\begingroup$ Why do you make point #3? $\endgroup$
    – Dave
    Oct 15, 2020 at 22:01
2
$\begingroup$

The following Python snippet demonstrates up-sampling, by sampling with replacement the instances of the class that are less in number(a.k.a minority class) in a data frame to solving the class imbalance problem,

import pandas as pd

# df is a data frame with FRAUD as the target column with classes 0 and 1.  
# There are more instances of class 0 than class 1 in the data frame df.  

# Separate majority and minority classes
df_majority = df.loc[df.FRAUD == 0].copy()
df_minority = df.loc[df.FRAUD == 1].copy()

# Upsample minority class
df_minority_upsampled = resample(df_minority,
                             replace=True,  # sample with replacement
                             n_samples=498551,  # to match majority class
                             random_state=123)  # reproducible results

# Combine majority class with upsampled minority class
df_upsampled = pd.concat([df_majority, df_minority_upsampled])

# Display new class counts
print(df_upsampled.FRAUD.value_counts())
$\endgroup$
4
  • $\begingroup$ Welcome to CV. Could you please explain what your code does? Indeed, it would help even to tell us what language it uses! (Presumably it's R, but not necessarily.) What do its variables mean? How are they defined? What do functions like pd.concat, which are not defined in base R, do? $\endgroup$
    – whuber
    Feb 17, 2020 at 14:54
  • $\begingroup$ @whuber, this is python and pd refers to the pandas package. $\endgroup$ Mar 17, 2020 at 17:04
  • $\begingroup$ It is better to state explicitly what software and packages you are using in your answer. But the important issue is to explain what the code is intended to do. $\endgroup$
    – whuber
    Mar 17, 2020 at 20:54
  • 2
    $\begingroup$ Plus, it's lifted directly from here (with a few minor edits); towardsdatascience.com/… (I thought it looked familiar). $\endgroup$ May 22, 2020 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.