I am dealing with a very unbalanced binary classification problem: 1% positives, 99% negatives. Training set is around 10 million rows, 40 columns. I choose the decision threshold (cutoff) on the training set as to match the number of positives in-sample. However, when I go out of sample, I underpredict the number of positives by around 20%. What are some things I can do to fix this problem? I would like to match the number of positives out of sample.
My AUC is decently high, around 94%, but the precision and recall are around 30% (these numbers hold both in-sample and out-of-sample). I am currently not using intercept in my model (to reduce risk of overfitting) - should I change this?
I am using binary cross entropy loss for training.
I don't know if I am missing something, perhaps to look into getting better features to improve precision and recall, or it would be hard to get a good match because the problem is so unbalanced? Would it make sense to try randomized (stochastic) threshold, would that even make sense or solve anything? What other ideas can I try?
Most of my features (columns) are discrete one-hot encodings, and I can clearly see that for some of them even in sample I am not matching the number of positives well, e.g. there's one bin that overestimates and another that underestimates, is there anything I can change related to the training (e.g. the loss function) or something else as to nudge the model to match the counts in each bin (predicted vs actual) during training?
Update: In some comments below, some people suggest I don't even need a threshold. Then, how do I obtain a binary 0 vs 1 decision, which is my ultimate goal, without using a threshold? What strategies and other approaches are there for the decision-making process?