I have a large dataset constituted of many ad impressions. My dependent binary variable clicked describe whether or not the ad was clicked on. As you can expect, the number of clicks is about 1000x smaller than the number of non-clicks in my dataset.
I'm fitting an Online Logistic Regression to this dataset and I found out that my predictions seem to be underestimating the observed click-through rate
King and Zeng (2002) claim that "Logistic Regression can sharply underestimate the probability of rare events". Firth (1993) proposed a preventive method to avoid the first order bias in Logistic Regression by using Jeffreys Prior:
Rather than applying the usual gradient: $$ U(\beta_r) = \sum_{i=1}^n (y_i - p_i)x_{ir} = 0$$
the following gradient should correct the first order bias: $$ U^*(\beta_r) = \sum_{i=1}^n (y_i - p_i + h_i(\frac{1}{2}-p_i))x_{ir} = 0$$
where:
- $y_i$ is the observed target for observation $i$
- $p_i$ is the model prediction for observation $i$
- $x_{ir}$ is the value of the feature $r$ for observation $i$
- $h_i$ is the $i$-th diagonal element of $H=W^{\frac{1}{2}}X(X^TWX)^{-1}X^TW^{\frac{1}{2}}$, with $W=diag(p_i(1-p_i))$
This formula, described by Heinze & Schemper (2002) too was used for the implementation of the logistf package in R.
As you can imagine, the calculation of $H$ for large datasets can be quite expensive. Hence my following questions:
- Did anyone tried to adapt the Firth Method to Online Logistic Regression? How did you simplify the calculation of $h_i$?
- Are there different approaches to correct the underestimating bias in online logistic regression for large unbalanced datasets?