# Why Is Jeffreys's Prior Used to Correct Biases?

Over here, a modification ("Firth's Correction") to the classical likelihood function has been proposed in which a penalty term has been added based on the square root of the Fisher Information. As we know, the square root of the Fisher Information is closely related to the Jeffreys's Prior:

$$\mathcal{L}(\theta) + \frac{1}{2} \log\left|\mathbf{I}(\beta)\right|$$

I am trying to understand the logic as to why a penalty term was chosen that was based on the Jeffreys's Prior and why exactly it is useful for correcting biases associated with rare events.

For instance, when it comes to Penalized Regression, I have read about penalty terms based on the L1 Norm and L2 Norms (e.g. LASSO and Ridge). Visually, I can understand why such penalty terms might be useful. The following types of illustrations demonstrate how such penalty terms serve to "push" regression coefficient estimates towards 0 and thereby might be able to mitigate problems associated with overfitting:

However, in the case of Firth's Correction, I am not sure as to how the square root of the Jeffreys's Prior is useful in correcting biases associated with rare events - mathematically speaking, how exactly is a penalty term based on the square root of the Jeffreys's Prior able to reduce biases associated with rare events? Currently, this choice of penalty seems somewhat arbitrary to me and I can't understand how it serves to reduce bias.

• As in the tag the prior is named for Harold Jeffreys. Acceptable spellings are Jeffreys prior or Jeffreys’ or Jeffreys’s. Commented Apr 22, 2023 at 6:31
• The hyperlink "Logistic Regression for Rare Events" actually links to the paper "Bias Reduction of Maximum Likelihood Estimates". A proper citation would not be amiss to avoid confusion. Commented Apr 22, 2023 at 23:17
• Found this answer which seems quite relevant: Seeking a Theoretical Understanding of Firth Logistic Regression Commented Apr 24, 2023 at 8:06
• On a related note -- and I know you've received this advice before -- you'll learn a lot from searching for and reading previous CV threads on topics of interest to you. As an experienced CV user, you of course know how to do this kind of research. Commented Apr 24, 2023 at 8:10
• @ dipetkov: thank you so much for your reply! This was actually one of the questions I had consulted earlier, but I was looking for more information on this topic. I can include a list of questions I consulted in the reference if you would like. thank you so much! Commented Apr 24, 2023 at 14:04

You are looking at the Firth correction through the lens of Jeffreys' prior, which isn't a helpful way of looking at it in this case. Instead, better to think of it as its own thing, based on the observed information matrix, and Jeffreys' prior as its own thing, based on the expected information matrix. The Firth correction isn't based on the Jeffreys' prior at all, it just happens to generate the same penalty term in the case of exponential families with canonical parameterization. The objectives of the two are different; consequently, the fact that they are sometimes the same mathematically is irrelevant to the raison d'etre of either.

Quoting from the abstract of Firth's paper (Bias reduction of maximum likelihood estimators):

It is shown how, in regular parametric problems, the first-order term is removed from the asymptotic bias of maximum likelihood estimates by a suitable modification of the score function. In exponential families with canonical parameterization the effect is to penalize the likelihood by the Jeffreys invariant prior.

Note that the adjustment involves the observed information matrix, not the expected information matrix; in exponential families with canonical parameterizations ("efwcp" for short,) the two are the same, but otherwise, not. Firth shows that the corrected estimator based on the observed information matrix is second-order unbiased but that the one based on the expected information matrix is not, except in the efwcp case. So, in the non-efwcp case, the Firth correction does not correspond to the Jeffreys' prior.

Firth's correction is not specific to logistic regression, but applies much more generally (as the abstract indicates,) and the paper has examples from Gaussian and Poisson regression as well. As you suspected, the Sivia chapter and Firth corrections are indeed related; the Sivia chapter illustrates a special case of the Firth correction, namely, Gaussian regression with a canonical link function / parameterization.

Firth's correction[1] is derived from an application of Cox and Snell (1968)[2] and McCullagh (1987)[3] analytical expressions for the first-order bias of maximum likelihood estimators to build a bias-correction procedure for parametric models in the exponential family.

Let $$\theta\in\Theta\subseteq\mathbb{R}^q$$ be the finite-dimensional parameter of a regular statistical model $$p(x\mid\theta)$$, and let $$\hat{\theta}$$ be its maximum likelihood estimator using $$n$$ samples. McCullagh (1987), based partially on Cox and Snell (1968), shows that the bias on the $$j$$-entry of $$\hat{\theta}$$, with $$j\in\{1,\cdots,q\},$$ is given by: $$b(\hat{\theta}_j) = \mathbb{E}[\hat{\theta}_j-{\theta}_j^*] = \underbrace{\frac{-1}{n}\sum_{s,t,u=1}^q k_{j,s}k_{t,u}(\kappa_{s,t,u}+\kappa_{s;t,u})/2}_{\text{first-order bias, } b_1(\hat{\theta}_j)\,\in\, O(n^{-1})} + O(n^{-2}),$$

where, $$k_{t,u}$$ is the $$(t,u)$$-entry of the inverse of the per-unit Fischer information matrix $$k(\theta)=i(\theta)^{-1}$$, with: $$i(\theta) = - \mathbb{E}\left[ \nabla_\theta^2 \log p(x\mid\theta)\right],$$

and $$\kappa_{s,t,u},\kappa_{s;t,u}$$ are joint null cumulants, as measures of per-unit information. If the total log-likelihood function is $$l_n(\theta)=\sum_{m=1}^n \log p(x_m\mid\theta)$$, then: \begin{aligned} \kappa_{s,t,u} &= n^{-1}\mathbb{E}\left[\frac{\partial l_n(\theta)}{\partial\theta_s}\cdot \frac{\partial l_n(\theta)}{\partial\theta_t}\cdot \frac{\partial l_n(\theta)}{\partial\theta_u}\right]\overset{iid}{=}-\left[\frac{\partial i(\theta)}{\partial\theta_s}\right]_{t,u},\\ \kappa_{s;t,u} &= n^{-1}\mathbb{E}\left[\frac{\partial l_n(\theta)}{\partial\theta_s}\cdot\frac{\partial^2 l_n(\theta)}{\partial\theta_t\partial\theta_u}\right]. \end{aligned}

Firth's bias correction procedure was initially motivated for parametric models in the exponential family. So, let us consider a distribution $$p$$ in this family, with canonical parameter $$\theta\in\Theta\subseteq\mathbb{R}^q$$. We can express the unit log-likelihood as: $$l(\theta)=\log p(x\mid\theta)=\log h(x)+\theta\cdot T(x) - A(\theta),$$

where $$T(x)$$ is the sufficient statistic. Let $$U(\theta)=\nabla_\theta l_n(\theta)=T(\vec{x})-n\nabla_\theta A(\theta)$$ be the total score function. The true parameter $$\theta^*$$ satisfies the moment condition $$\mathbb{E}[U(\theta^*)]=\vec{0}$$, and the maximum likelihood estimator $$\hat{\theta}_{M}$$ is defined as the solution of $$U(\hat{\theta})=\vec{0}$$. So, under correct model specification, the bias in $$\hat{\theta}$$ comes from bias in the estimated $$U$$ when using finite data, particularly for cases of small sample size and/or rare events. Notice that the sufficient statistic $$T$$ enters linearly in $$U$$, so it only changes its location but not its shape. Thus, the bias coming from the data would translate the whole function $$U$$ upwards or downwards.

Then, for a small bias $$b(\theta)=\Delta\theta$$, the associated change from $$U$$ to the translated $$\tilde{U}$$ is: $$\tilde{U}(\theta)-U(\theta)\approx \nabla_\theta U(\theta)\cdot \Delta\theta = -I_o(\theta)\cdot b(\theta),$$

where $$I_o(\theta)=-\nabla_\theta U(\theta)=-\nabla_\theta^2 l_n(\theta)$$ is the total observed information matrix, but the total Fisher information $$ni(\theta)$$ can be used instead[4]. Then, we can express: $$\tilde{U}(\theta) = U(\theta) - ni(\theta)\cdot b(\theta).$$

Now, by (1) presenting the expressions at the $$j$$-entry level, (2) replacing $$b(\theta)$$ by the negative fist-order bias of the maximum likelihood estimator, and (3) noticing that $$\kappa_{s;t,u}=0$$ for all $$s,t,u\in\{1,\cdots,q\}$$ in exponential family models; Firth presents a modification of the score of $$j$$-entry of $$\theta$$ given by: \begin{aligned} \tilde{U}_j(\theta) &= {U}_j(\theta) - n\sum_{r=1}^q i_{j,r}[-b_1(\hat{\theta}_r)],\\ &= {U}_j(\theta) - n\sum_{r=1}^q i_{j,r}\left[\frac{1}{n}\sum_{s,t,u=1}^q k_{r,s}k_{t,u}(\kappa_{s,t,u}+0)/2 \right],\\ &= {U}_j(\theta) - \frac{1}{2}\sum_{r=1}^q i_{j,r}\sum_{s,t,u=1}^q k_{r,s}k_{t,u}\kappa_{s,t,u},\\ &= {U}_j(\theta) - \frac{1}{2}\sum_{t,u=1}^q k_{t,u}\kappa_{j,t,u} = {U}_j(\theta) - \frac{1}{2}\operatorname{trc}\left[k(\theta)\cdot \left(-\frac{\partial i(\theta)}{\partial\theta_j}\right)\right],\\ &= {U}_j(\theta) + \frac{1}{2}\operatorname{trc}\left[i^{-1}(\theta)\cdot \left(\frac{\partial i(\theta)}{\partial\theta_j}\right)\right]={U}_j(\theta) + \frac{1}{2}\frac{\partial}{\partial\theta_j}\log\det i(\theta), \end{aligned}

where the equivalences come from the algebra of null joint cumulants, the relationship of $$\kappa_{j,t,u}$$ with the derivatives of the information matrix $$i(\theta)$$, and from the fact that for a invertible matrix $$A$$, we have that $$\frac{d}{dx}\log\det A=\operatorname{trc}\left(A^{-1}\frac{d}{dx}A\right)$$.

$$\tilde{U}_j(\theta)$$ is clearly the $$j$$-entry of the gradient of the modified log-likelihood: $$\tilde{l}_n(\theta) = {l}_n(\theta) + \frac{1}{2}\log\det i(\theta),$$

Hence, $$\tilde{l}_n(\theta)$$ is the corrected/penalized log-likelihood required to correct the first-order bias of the maximum likelihood estimator. It just happens to match the results from applying a Jeffreys's prior in Bayesian analysis.

References and notes

[1] FIRTH, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika, 80(1), 27–38. doi:10.1093/biomet/80.1.27

[2] COX, D. R. and SNELL, E. J.(1968). A General Definition of Residuals. Journal of the Royal Statistical Society: Series B (Methodological), 30(2), 248-265. doi:10.1111/j.2517-6161.1968.tb00724.x

[3] McCULLAGH, P. (1987). Tensor methods in statistics. Chapman and Hall. London.

[4] In the amendments and corrections of Biometrika (1995), he states that either choice is second-order efficient, and thus equivalent.

• From your answer I surmise that Firth's bias correction "accidentally" coincides with the logistic MAP from Jeffreys prior, but doesn't the fact that this bias correction also emerges in the "normal MAP from Jeffreys prior" scenario (mentioned but not explained in Sivia, pp 52-3) hint at some deeper phenomenon? Commented Jul 7 at 13:47