Logistic regression with rare events, all events have large residuals and are influential points

Question

I am running a logistic regression model where the outcome is relatively rare (250 out of 5000). My main interest is to see if there are differences between age groups, sex, educational levels, income levels...with regards to the outcome and i am running a model with just the main effects. It appears that since the model has so few "positive" outcomes, these are not fitted well by the model. This becomes apparent since all the standardized residuals for these cases are between 2-3.1 and that all these are seen as influential points based on Cook's distance (see graph), based on the common rule of thumb 4/N(=0.0008). This is not surprising since the highest predicted probability from the model is p=0.28 for one of the "positive" outcomes.

My question is if such a model is trustworthy. I am only interested in the obtained odds ratios, not how well the model can discriminate between positive/negative outcomes. I have used the Le Cessie-van Houwelingen-Copas-Hosmer global goodness of fit test to evaluate the model which gives a p-value of p-value=0.3, so from that prespective looks good but on the other hand this is expected since the vast majority of the sample is well predicted. I have also tried Firth's logistic regression to correct for bias and both regressions give almost identical results. I have finally checked what happens if i remove the 10 more influential points from the model which resulted in almost identical estimated coefficients as the model with all data.

My question is if one can use a model like this for inference, for example to identify differences between age groups and being able to rely on these estimates.

I should also add that the model has 6 categorical predictors which leads to 20 model parameters. One variable is binary, 2 variables have four levels, 2 variables have 5 levels and one has 6 levels. No separation problems were observed and no multicollinearity based on VIF.

Answer 1

Those measures are not the preferred ones for assessing lack of fit in binary logistic regression, when compared to directed tests of linearity or additivity. Your effective sample size is about 500 and this sample size does not support consideration and estimation of very many parameters. With a relatively small sample size expect individual observations to be influential, but don't necessarily make adjustments because of that. And as @EdM mentioned the Efron-Gong optimism bootstrap can quantify overfitting/model reliability. Details about these issues are in RMS.

Answer 2

Most of the information in a logistic regression comes from the cases in the lower-prevalence category. Thus the "rule of thumb" for avoiding overfitting in a logistic regression is to have 10-20 lower-prevalence cases per model parameter evaluated. Under that "rule of thumb" you are on the low end of the range, but OK. That's unlike ordinary least squares, where the corresponding rule of thumb is for the total number of cases.

I'm not familiar with your "rule of thumb" about 4/N for Cook's distance. If that comes from ordinary least squares, it's possible that in the logistic-regression context the N should be the number of cases in the lower-prevalence category rather than the total number of cases. That would give you a "rule of thumb" for excess influence of 0.016, with all of your cases less than half that level.

Instead of relying on rules of thumb and diagnostic plots, let the data tell you how reliable your modeing process is. Repeat your modeling process on multiple bootstrapped samples of the data, examine the variance of your coefficient estimates among the multiple models (for inference), and evaluate how well the models work on the full data set. Under the bootstrap principle, that's a good way to estimate how well your model would work if applied to a new data sample.