I've a database with about 750000 observations, of which only 700 have a value of 1 for the response variable. Therefore, I've done some research on the issues occurring when estimating a logit with rare events. The following links contain useful information, by the way: https://statisticalhorizons.com/logistic-regression-for-rare-events
I know that R has a
brglm package which allows implementation of penalized maximum likelihood to reduce the bias (following Firth (1993)), but I'm more interested right now in being sure that I understand why the bias occurs in the first place.
So, from my understanding, the main problem here is separability. Perfect separability occurs when some continuous predictor is always associated with a response of 1 below some threshold, and never above, or when some value of a categorical predictor is always associated with a 1 while the other values never are. In such a case, knowing the value of the predictor is enough to predict perfectly the value of the response. As a result, the likelihood function can be made infinite by increasing or decreasing the predictor's coefficient to infinity. This, in practice, yields biased and unbelievably large coefficients.
That problem is more prevalent for rare outcomes precisely because we may have so few 1's that they all fall in a predictor's tail, or in some discrete class. With a bigger dataset, we would eventually get 1's outside of these areas, and as a consequence the "perfect prediction" is spurious. Yet, precisely because 1's are slow to come, this may not have occurred for the data at hand.
Am I getting it right? And are there other problems with using ML to estimate log-odds of rare events?
I'll take an extreme example. Assume we're interesting in estimating the correlation between religion and the probability of having completed college, and that in the society we study, college completion is extremely rare. In fact, in our dataset, there's only one college graduate. Turns out this guy reported affiliation to the Jedi religion, and that he's the only Jedi in the dataset. Then, knowing whether a person is a Jedi or not is enough to predict college completion with no fault... according to our limited dataset. Hence the perfect prediction problem.
Now, assume we remove "Jedi religion" as a separate class and bundle it with an encompassing "Other" class, along with other little represented religions. This class will then feature both values of the response, depending on which person you look at. Therefore, we have solved the (perfect) separability issue.
More generally, am I right to assume that, in the case of categorical predictors, the failures of rare events logits can be alleviated by using broader categories to avoid separability? Likewise, in the case of a continuous variables, if there was an equivalence between the set of 1's and the set of values of the predictor above or below some threshold, we could alleviate the problem by binning the continuous variable into broad enough ordered categories.
What, then, are the remaining issues with rare events logits? Assuming the structure of our data is simple enough that I can simply tabulate confusion matrices and reorganize categories if needs be, why would I need to run more complex estimation procedures than basic ML?
Thanks for your feedback.