How to improve F1 score with skewed classes?

Question

I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1.

I've tried several classification algorithms, without too much fine tuning, just to get a feeling of how the baseline performance was. They all got an accuracy score of around 99%, that is exactly the ratio between class 0 samples and total samples. Artificially under-sampling just got the accuracy score down to the very same ratio of the new dataset, so no improvement on that side.

The F1 score is really bad because I'm experiencing awful Type II errors: basically, the algorithm is just guessing that everything is belonging to class 0. With some models that I tried, it literally predicts everything to be class 0: false positives are 0 (because no positive samples get predicted) and false negatives are really a lot (because actual positives are predicted to be negative). The AUC-ROC is around 50% (awful), and weighting the models to take into account the skewness of the classes brought no improvement.

I tried to do some feature engineering (ensembling the supervised classification on top of some unsupervised clustering), with almost no luck.

Do you have any suggestions regarding how to tackle such problems / how to diagnose the underlying issue(s) that prevent(s) the predictor from being accurate? Or I should take this as a proof that, given my the dataset, belonging to class 1 is just random (so I should collect more features)?

Side note: I thought about taking it from another side, ie. anomaly detection, but I'm not sure this could be the right approach.

Answer 1

Most of the classification problems I've tackled are similar in nature, so a large class imbalance is quite common.

It is not clear whether you are using training-validation sets to build and fine tune the model. Cross-fold validation is generally preferred since it gives more reliable model performance estimates.

The F1 score is a good classification performance measure, I find it more important than the AUC-ROC metric. Its best to use a performance measure which matches the real-world problem you're trying to solve.

Without having access to the dataset, I'm unable to give exact pointers; so I'm suggesting a few directions to approach this problem and help improve the F1 score:

1. Use better features, sometimes a domain expert (specific to the problem you're trying to solve) can give relevant pointers that can result in significant improvements.

2. Use a better classification algorithm and better hyper-parameters.

3. Over-sample the minority class, and/or under-sample the majority class to reduce the class imbalance.

4. Use higher weights for the minority class, although I've found over-under sampling to be more effective than using weights.

5. Choose an optimal cutoff value to convert the continuous valued class probabilities output by your algorithm into a class label. This is as important as a good AUC metric but is overlooked quite often. A word of caution though: the choice of the cutoff should be guided by the users by evaluating the relevant trade-offs.

Answer 2

The following Python snippet demonstrates up-sampling, by sampling with replacement the instances of the class that are less in number(a.k.a minority class) in a data frame to solving the class imbalance problem,

import pandas as pd

# df is a data frame with FRAUD as the target column with classes 0 and 1.  
# There are more instances of class 0 than class 1 in the data frame df.  

# Separate majority and minority classes
df_majority = df.loc[df.FRAUD == 0].copy()
df_minority = df.loc[df.FRAUD == 1].copy()

# Upsample minority class
df_minority_upsampled = resample(df_minority,
                             replace=True,  # sample with replacement
                             n_samples=498551,  # to match majority class
                             random_state=123)  # reproducible results

# Combine majority class with upsampled minority class
df_upsampled = pd.concat([df_majority, df_minority_upsampled])

# Display new class counts
print(df_upsampled.FRAUD.value_counts())