I have a large dataset constituted of many ad impressions. My dependent binary variable clicked describe whether or not the ad was clicked on. As you can expect, the number of clicks is about 1000x smaller than the number of non-clicks in my dataset.

I'm fitting an Online Logistic Regression to this dataset and I found out that my predictions seem to be underestimating the observed click-through rate

King and Zeng (2002) claim that "Logistic Regression can sharply underestimate the probability of rare events". Firth (1993) proposed a preventive method to avoid the first order bias in Logistic Regression by using Jeffreys Prior:

Rather than applying the usual gradient: $$ U(\beta_r) = \sum_{i=1}^n (y_i - p_i)x_{ir} = 0$$

the following gradient should correct the first order bias: $$ U^*(\beta_r) = \sum_{i=1}^n (y_i - p_i + h_i(\frac{1}{2}-p_i))x_{ir} = 0$$


  • $y_i$ is the observed target for observation $i$
  • $p_i$ is the model prediction for observation $i$
  • $x_{ir}$ is the value of the feature $r$ for observation $i$
  • $h_i$ is the $i$-th diagonal element of $H=W^{\frac{1}{2}}X(X^TWX)^{-1}X^TW^{\frac{1}{2}}$, with $W=diag(p_i(1-p_i))$

This formula, described by Heinze & Schemper (2002) too was used for the implementation of the logistf package in R.

As you can imagine, the calculation of $H$ for large datasets can be quite expensive. Hence my following questions:

  1. Did anyone tried to adapt the Firth Method to Online Logistic Regression? How did you simplify the calculation of $h_i$?
  2. Are there different approaches to correct the underestimating bias in online logistic regression for large unbalanced datasets?
  • $\begingroup$ Have you tried regularization? It might help. $\endgroup$
    – SamParker
    Jan 1, 2016 at 18:19

1 Answer 1


First, I must admit I don't know exactly what you mean by 'online' logistic regression. Of course, calculation of H is expensive if you really do the matrix operations. However, all that is needed is the diagonal elements of H, who come at much lower cost. Depending on your explanatory variables, you may be able to group your data such that each covariable/outcome combination can be assigned a frequency count. This speeds up calculations dramatically. Both these options are implemented (and used by default) in the current version of our R package logistf.

  • $\begingroup$ By "online" I'm assuming the OP is referring to some online ML package such as Mahout. Another potentially less computationally expensive approach would be to fit a poisson model. The poisson is a distribution that fits rare events and the extremes of a distribution much better than the logistic. $\endgroup$
    – user78229
    Oct 14, 2015 at 15:52
  • $\begingroup$ See Wikipedia for a definition of online algorithms. $\endgroup$
    – Andy W
    Oct 14, 2015 at 17:07
  • $\begingroup$ The Poisson distribution is for counts, not for binary $Y$. $\endgroup$ Jan 1, 2016 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.