I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1.
I've tried several classification algorithms, without too much fine tuning, just to get a feeling of how the baseline performance was. They all got an accuracy score of around 99%, that is exactly the ratio between class 0 samples and total samples. Artificially under-sampling just got the accuracy score down to the very same ratio of the new dataset, so no improvement on that side.
The F1 score is really bad because I'm experiencing awful Type II errors: basically, the algorithm is just guessing that everything is belonging to class 0. With some models that I tried, it literally predicts everything to be class 0: false positives are 0 (because no positive samples get predicted) and false negatives are really a lot (because actual positives are predicted to be negative). The AUC-ROC is around 50% (awful), and weighting the models to take into account the skewness of the classes brought no improvement.
I tried to do some feature engineering (ensembling the supervised classification on top of some unsupervised clustering), with almost no luck.
Do you have any suggestions regarding how to tackle such problems / how to diagnose the underlying issue(s) that prevent(s) the predictor from being accurate? Or I should take this as a proof that, given my the dataset, belonging to class 1 is just random (so I should collect more features)?
Side note: I thought about taking it from another side, ie. anomaly detection, but I'm not sure this could be the right approach.