Wrong predictions with imbalanced binary data

Question

I have a sample of 3000 observations. I want to study the impact of covariates on a binary dependent variable (i.e. two categories: "yes" or "no") using either a logistic regression or a linear probability model. The sample is imbalanced (95% "yes", 5% "no"). The two models have pretty bad predictions (i.e. they do not predict the "no's" at all) and very low R2 (or pseudo R2 for the logistic regression).

I understand that wrong predictions can be due to the imbalanced sample, but what about the low R2/pseudo R2, can it also be due to the imbalanced sample?

What are the advantages/disavantages of using one of these two models when dealing with rare events? What could I do to have better predictions (or maybe measure predictions of these models more accurately?

Answer 1

Your notion of 'prediction' is incorrect. For binary regression, the prediction is the predicted probability that Y=1 | X. Imbalance causes no problem in getting this prediction. Imbalance just makes standard errors of individual regression coefficients higher but that's to be expected.

More details about classification vs. prediction may be found here, and related problems caused by improper accuracy scoring rules are described here.

Your sample is too small (by far) for split sample validation to work, i.e., to be stable. Use the Efron-Gong "optimism" bootstrap. See for example the lrm, validate, and calibrate functions in the R rms package.

Answer 2

When dealing with highly imbalanced datasets, go for AUC and f1 score of each class as metrics. Also, use confusion matrix to checkout what's actually going on behind the validation accuracy. One more thing, make sure you are using stratified sampling for splitting the dataset into train:test or train:valid:test, scikit-learn's train_test_split does it by default. Otherwise, you might miss out the classes.

R2 isn't enough,

R-squared does not indicate whether a regression model is adequate. You can have a low R-squared value for a good model, or a high R-squared value for a model that does not fit the data!

source for the above quote.

Hope it helps.