Match number of positives in unbalanced data set

Question

I am dealing with a very unbalanced binary classification problem: 1% positives, 99% negatives. Training set is around 10 million rows, 40 columns. I choose the decision threshold (cutoff) on the training set as to match the number of positives in-sample. However, when I go out of sample, I underpredict the number of positives by around 20%. What are some things I can do to fix this problem? I would like to match the number of positives out of sample.

My AUC is decently high, around 94%, but the precision and recall are around 30% (these numbers hold both in-sample and out-of-sample). I am currently not using intercept in my model (to reduce risk of overfitting) - should I change this?

I am using binary cross entropy loss for training.

I don't know if I am missing something, perhaps to look into getting better features to improve precision and recall, or it would be hard to get a good match because the problem is so unbalanced? Would it make sense to try randomized (stochastic) threshold, would that even make sense or solve anything? What other ideas can I try?

Most of my features (columns) are discrete one-hot encodings, and I can clearly see that for some of them even in sample I am not matching the number of positives well, e.g. there's one bin that overestimates and another that underestimates, is there anything I can change related to the training (e.g. the loss function) or something else as to nudge the model to match the counts in each bin (predicted vs actual) during training?

Thank you

Update: In some comments below, some people suggest I don't even need a threshold. Then, how do I obtain a binary 0 vs 1 decision, which is my ultimate goal, without using a threshold? What strategies and other approaches are there for the decision-making process?

Answer 1

Your question indicates a number of common errors in regression modeling.

1. Omission of the intercept in order to have fewer parameters and less overfitting

Particularly in an imbalanced problem where you want the model to be skeptical about membership in the minority class, the intercept is important. I understand the idea of model parsimony in hopes of avoiding overfitting, but the intercept plays a unique role in the model by affecting the outcome in the same way no matter what the features are.

No other parameter in the model can do this, so if you're going to do away with a parameter, that isn't the one you want to lose unless you have an excellent reason for doing so (such as knowing that the outcome is zero when the features all are zero).

2. Pure linearity

You don't state anything about this in the question, but you are allowed to have nonlinear features like splines in your generalized linear model. By allowing the probability to increase, decrease, and then increase again (for instance) as a feature increases, you allow for better predictions than you would get from forcing the straight-line fit that I suspect you are using.

3. Parsimony

In one of his videos, the same Frank Harrell as in the comments mentions that parsimony is the enemy of predictive accuracy. My answer here gets into some of why that might be the case and links to additional material (by Frank Harrell). If you're worried about overfitting but want to have a lot of variables, interactions, and nonlinear features (e.g., splines), use some kind of penalized estimation, such as a ridge penalty. This fights overfitting without getting rid of any of the parameters.

All of these and more are discussed in Frank Harrell's Regression Modeling Strategies textbook.

Finally, by having a good model that considers all of these issues, you can, if you must, apply a threshold to make dichotomous decisions based on the known costs of mistakes, and that is how you should pick your threshold. I would argue that, if you don't know the costs of your mistakes, you have no business making classifications and should only give predicted probabilities.

In the extreme, where it is unacceptable to mistake a $0$ for a $1$ or a $1$ for a $0$, the best dichotomous decision might be to classify every instance as one class so that you never make the unacceptable mistake.

Additionally, don't be surprised when you find yourself unable to make accurate dichotomous decisions for instances whose predicted probabilities are near the threshold. The best decision in such a situation might be no decision. This is related to what Kolassa means when he writes that "a cost optimal decision might also well include more than one threshold!"

If I may pontificate at the end of my answer, your post suggests that you face a common issue for data scientists to face. You don't have to knuckle under to every bad idea someone has. You're the expert; act like it. This doesn't mean being an arrogant punk, but you're allowed to explain why someone is wrong. This is scary the first time you have to do it, and it is when you are most valuable as a statistician or regression expert. (It also get less scary as you do it more and more.)