Logistic regression and discrepant sample sizes between 0/1 groups

Question

I currently working in a multivariate logistic model but I have a problem regarding the sample size of my observations:

-The "success" (1) event group has a sample size of 249 distinct observations - The "non success" (0) event group has a sample size of 48,957, and it's a significant part of the population, and many times larger than the "success" group.

So when I fit a multivariate logistic regression model, two things happen: - Because of the big sample size, The p-values of the model coefficients become always significant at alpha 1% - The fitted predicted probability variation between the groups becomes very tiny, even if the independent variables are good predictors.

So I was suggested making a bootstrap of the model. So here is my doubt :

-Should I do it the more usual way, taking smaller arbitrary size random samples of the complete sample (both groups) with replacement ( or without replacement in this case?)

OR

-Should I keep the the small "success" group constant and them add an equal number of different "non success" randomly picked cases in each sample. Is this "cheating"?

OR

-Should I just stay with the original whole sample model, and bear with it's little predictive power (less than 1% fitted predicted difference between the groups ), even when using significant predictors that differentiate the groups.

In both cases this is not exactly the usual bootstrap as I wish to make sub-samples of a larger group, the original big sample, and perhaps what I want is not bootstrap at all. My idea is to minimize the discrepancy between the two groups sizes, the inflated p-values, and the small probability fitted difference between groups due to tho fact that the "success" event is rare in my sample. Is this theoretically correct?

The final objective is to obtain an "avarage" model with the mean coefficients and use it to calculate the probability of the "non success" cases actualy being "successes"

Or does anybody has other idea for this question? Feel free to post any suggestion in R language. I know that there is a similar question but I want to know also if my ideas about this are wrong.

Answer 1

It sounds like your problem is analogous to "Predict who will win the lottery based on whether they buy a lottery ticket." Now, whether someone buys lottery tickets is a very significant predictor of whether they'll win the lottery (indeed it's basically the only predictor there is!) By buying a lottery ticket their odds of winning increase from 0% to some small chance (perhaps .001% for a small lottery).

However, the fact that this predictor is very significant does not mean you can successfully predict who wins the lottery! All you've done is go from a probability of 0 to a higher, but still very low, probability. In this scenario there is no method that can fix this problem. (If you come up with a way to reliably predict lottery winners in advance, please let me know!)

It is still possible that you have a predictor that predicts well even if the overall rate of successes is very low: for instance, "predict who will lose the lottery this week based on who bought a ticket." Imagine 1 million Americans (out of 300 million) buy a ticket for a particular lottery, and only one of them will win. That means that there are 1 million - 1 people who played the lottery and lost (only about .3% of the country, a low proportion). But if you divide it based on who bought a ticket, it turns into "If you bought a ticket, you have a 99.9999% chance of losing the lottery, compared to a 0% chance if you didn't buy one"- quite a good predictor!

However, if that kind of predictor does exist in your data, logistic regression would certainly show a large increase in probability predicted. Therefore, it sounds like this is not the case.

For more about these issues, see Logistic Regression in Rare Events Data.