4
$\begingroup$

I am trying to learn more about how to apply Logistic Regression in the presence of rare events.

I am trying to understand what is written in this paper here: https://gking.harvard.edu/files/0s.pdf

If I understand correctly, in the presence of rare events, the MLE estimate for the intercept of a Logistic Regression can be updated using the following correction formula (where $tau$ represents the true proportion of events within the population):

$$\hat{\beta}_0 - \ln\left[\left(1 - \frac{\tau}{\tau}\right)\cdot\left(\frac{\bar{y}}{1 - \bar{y}}\right)\right]$$

Apparently, using this corrected formula can provide more reasonable estimates for the value of $\hat{\beta}_0$ and the variance of $\hat{\beta}_0$.

I had the following question: In this paper, there is no correction formula provided for the estimates of $\hat{\beta}_1$ - is there any reason for this? In the presence of rare events, is the classical MLE estimate for $\hat{\beta}_1$ somehow less affected compared to $\hat{\beta}_0$?

Thanks!

$\endgroup$

2 Answers 2

9
$\begingroup$

To start with, you have the equation wrong. The bias correction is not $$\log\left[\left(1-\frac{\tau}{\tau}\right)\left(\frac{\bar y}{1-\bar y}\right)\right],$$ it's $$\log\left[\left(\frac{1-\tau}{\tau}\right)\left(\frac{\bar y}{1-\bar y}\right)\right].$$

This not a bias correction for rare events generally (like the Firth correction). It's a bias correction specifically logistic regression and for case-control sampling: if you oversample individuals with $y=1$ relative to those with $y=0$. And yes, this bias is only in the intercept -- a surprising and important fact.

The bias being only in the intercept is unique to case-control sampling and unique to models for the odds ratio such as logistic regression. It's one of the reasons logistic regression has been so popular in epidemiology.

$\endgroup$
3
  • $\begingroup$ I think it's important to stress that the underlying assumption is that any over- or undersampling must be independent of the independent variable. $\endgroup$
    – Luca Citi
    Commented Apr 21, 2023 at 10:23
  • $\begingroup$ Sampling is absolutely not going to be independent of $x$. What is important is that it's independent of $x$ conditional on $y$. $\endgroup$ Commented Apr 22, 2023 at 3:15
  • $\begingroup$ I assumed that to be implicit in the concept of under-/oversampling, e.g. oversampling "individuals with y=1 relative to those with y=0" must be independent of X. The conditionality on Y is already in the quoted part (which is in your text). Anyway, put it as you want but I think it's essential that this (conditional) independence on X is part of your answers, because that's really what allows us to claim that only $\beta_0$ is affected. $\endgroup$
    – Luca Citi
    Commented Apr 22, 2023 at 6:00
4
$\begingroup$

Imagine an intercept-only logistic regression model

$$ P(y=1) = E[y] = g^{-1}(\beta_0) $$

In such a model, $\beta_0$ transformed via the logistic function $g^{-1}$ is the mean of $y$. If we add a parameter $x$ to the model, it becomes

$$ P(y=1|x) = E[y|x] = g^{-1}(\beta_0 + \beta_1 x) $$

Now $\beta_0$ would correct the bias of the model. Recall that by bias we would mean here that $E[y] \ne E[g^{-1}(\beta_0 + \beta_1 x)]$. Logistic regression is a linear model, so let's take one step back to linear equations for a while. In a linear equation, $ax + b$ changing $a$ would shift the line vertically (see the image below taken from here).

Two linear models: 3/4x - 3 and 3/4x + 1.

In the regression case, shifting it over the $y$ axis is used for correcting the bias since it changes the expected value of the predictions for $y$ by a constant for it to match with $E[y]$.

In the base rate correction, we are correcting for the fact that the base rate in the population that you are making the predictions for differs from the sample that was used to fit your model assuming that the relationship between $x$ and $y$ is the same in both. We are not aiming to correct $\beta_1$, nor do we assume that it differs.

Now imagine that we "corrected" $\beta_1$. This would lead to a completely different model! You could change the parameters to arbitrary values, but then you don't need any data at all, you can just make up the parameters and plug-in into the equation. If you assume that the relationship between $x$ and $y$ differs, then you need to fit a model on a new dataset that comes from the same population as the population you want to make the predictions for.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.