4
$\begingroup$

I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1.

I've tried several classification algorithms, without too much fine tuning, just to get a feeling of how the baseline performance was. They all got an accuracy score of around 99%, that is exactly the ratio between class 0 samples and total samples. Artificially under-sampling just got the accuracy score down to the very same ratio of the new dataset, so no improvement on that side.

The F1 score is really bad because I'm experiencing awful Type II errors: basically, the algorithm is just guessing that everything is belonging to class 0. With some models that I tried, it literally predicts everything to be class 0: false positives are 0 (because no positive samples get predicted) and false negatives are really a lot (because actual positives are predicted to be negative). The AUC-ROC is around 50% (awful), and weighting the models to take into account the skewness of the classes brought no improvement.

I tried to do some feature engineering (ensembling the supervised classification on top of some unsupervised clustering), with almost no luck.

Do you have any suggestions regarding how to tackle such problems / how to diagnose the underlying issue(s) that prevent(s) the predictor from being accurate? Or I should take this as a proof that, given my the dataset, belonging to class 1 is just random (so I should collect more features)?

Side note: I thought about taking it from another side, ie. anomaly detection, but I'm not sure this could be the right approach.

$\endgroup$
7
$\begingroup$

Most of the classification problems I've tackled are similar in nature, so a large class imbalance is quite common.

It is not clear whether you are using training-validation sets to build and fine tune the model. Cross-fold validation is generally preferred since it gives more reliable model performance estimates.

The F1 score is a good classification performance measure, I find it more important than the AUC-ROC metric. Its best to use a performance measure which matches the real-world problem you're trying to solve.

Without having access to the dataset, I'm unable to give exact pointers; so I'm suggesting a few directions to approach this problem and help improve the F1 score:

  1. Use better features, sometimes a domain expert (specific to the problem you're trying to solve) can give relevant pointers that can result in significant improvements.

  2. Use a better classification algorithm and better hyper-parameters.

  3. Over-sample the minority class, and/or under-sample the majority class to reduce the class imbalance.

  4. Use higher weights for the minority class, although I've found over-under sampling to be more effective than using weights.

  5. Choose an optimal cutoff value to convert the continuous valued output of your algorithm into a class label. This is as important as a good AUC metric but is overlooked quite often.

$\endgroup$
  • 2
    $\begingroup$ It is important that your learning algorithm predict class probabilities, and that a threshold is set for hard classification based on the risks of false negatives and false positives. I do not think your answer is complete without mentioning this. $\endgroup$ – Matthew Drury Apr 20 '17 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.