Bias Correction for Large Scale Logistic Regression with Rare Events

Question

I have a large dataset constituted of many ad impressions. My dependent binary variable clicked describe whether or not the ad was clicked on. As you can expect, the number of clicks is about 1000x smaller than the number of non-clicks in my dataset.

I'm fitting an Online Logistic Regression to this dataset and I found out that my predictions seem to be underestimating the observed click-through rate

King and Zeng (2002) claim that "Logistic Regression can sharply underestimate the probability of rare events". Firth (1993) proposed a preventive method to avoid the first order bias in Logistic Regression by using Jeffreys Prior:

Rather than applying the usual gradient: $$ U(\beta_r) = \sum_{i=1}^n (y_i - p_i)x_{ir} = 0$$

the following gradient should correct the first order bias: $$ U^*(\beta_r) = \sum_{i=1}^n (y_i - p_i + h_i(\frac{1}{2}-p_i))x_{ir} = 0$$

where:

• $y_i$ is the observed target for observation $i$
• $p_i$ is the model prediction for observation $i$
• $x_{ir}$ is the value of the feature $r$ for observation $i$
• $h_i$ is the $i$-th diagonal element of $H=W^{\frac{1}{2}}X(X^TWX)^{-1}X^TW^{\frac{1}{2}}$, with $W=diag(p_i(1-p_i))$

This formula, described by Heinze & Schemper (2002) too was used for the implementation of the logistf package in R.

As you can imagine, the calculation of $H$ for large datasets can be quite expensive. Hence my following questions:

1. Did anyone tried to adapt the Firth Method to Online Logistic Regression? How did you simplify the calculation of $h_i$?
2. Are there different approaches to correct the underestimating bias in online logistic regression for large unbalanced datasets?

Answer 1

First, I must admit I don't know exactly what you mean by 'online' logistic regression. Of course, calculation of H is expensive if you really do the matrix operations. However, all that is needed is the diagonal elements of H, who come at much lower cost. Depending on your explanatory variables, you may be able to group your data such that each covariable/outcome combination can be assigned a frequency count. This speeds up calculations dramatically. Both these options are implemented (and used by default) in the current version of our R package logistf.