16
$\begingroup$

I am trying to understand Firth logistic regression (method of handling perfect/complete or quasi-complete separation in logistic regression) so I can explain it to others in simplified terms. Does anyone have a dummied-down explanation of what modification Firth estimation is making to MLE?

I have read, as best I could, Firth (1993) and I understand a correction is being applied to the score function. I am fuzzy on the origin and justification of the correction and what role the score function plays in MLE.

Sorry if this is rudimentary knowledge. The literature I've reviewed seems to require a much deeper understanding of MLE than I possess.

$\endgroup$
13
$\begingroup$

Firth's correction is equivalent to specifying Jeffrey's prior and seeking the mode of the posterior distribution. Roughly, it adds half of an observation to the data set assuming that the true values of the regression parameters are equal to zero.

Firth's paper is an example of a higher order asymptotics. The null order, so to say, is provided by the laws of large numbers: in large samples, $\hat \theta_n \approx \theta_0$ where $\theta_0$ is the true value. You may have learned that MLEs are asymptotically normal, roughly because they are based on nonlinear transformations of sums of i.i.d. variables (scores). This is the first order approximation: $\theta_n = \theta_0 + O(n^{-1/2}) = \theta_0 + v_1 n^{-1/2} + o(n^{-1/2})$ where $v_1$ is a normal variate with zero mean and variance $\sigma_1^2$ (or var-cov matrix) that is the inverse of Fisher information for single observation. The likelihood ratio test statistic is then asymptotically $n(\hat\theta_n - \theta_0)^2/\sigma_1^2 \sim \chi^2_1$ or whatever the multivariate extensions to inner products and inverse covariance matrices would be.

Higher order asymptotics tries to learn something about that next term $o(n^{-1/2})$, usually by teasing out the next term $O(n^{-1})$. That way, the estimates and test statistics can incorporate the small sample biases of the order of $1/n$ (if you see the paper that says "we have unbiased MLEs", these people probably don't know what they are talking about). The best known correction of this kind is Bartlett's correction for likelihood ratio tests. Firth's correction is of that order, too: it adds a fixed quantity $\frac12 \ln \det I(\theta)$ (top of p. 30) to the likelihood, and in large samples the relative contribution of that quantity disappears at the rate of $1/n$ dwarfed by the sample information.

$\endgroup$
3
  • 1
    $\begingroup$ Sorry for my lack of understanding, but I'm not completely following. When you say "Roughly, it adds half of an observation to the data set assuming that the true values of the regression parameters are equal to zero." Why would you assume the true values of regression parameters are equal to zero? Also, how is it adding half an observation to the dataset? $\endgroup$ – ESmith5988 Mar 4 '14 at 17:59
  • 1
    $\begingroup$ From the rest of your explanation, it seems the likelihood function is being adjusted by a fixed quantity which reduces the positive bias of small samples. The fixed quantity is effectively a function of the information which goes to zero as sample size increases, correct? $\endgroup$ – ESmith5988 Mar 4 '14 at 17:59
  • 1
    $\begingroup$ On your first comment -- Firth correction is roughly the expected value of a contribution to the likelihood that would have been added by an observation that would have an effective weight of 1/2. This is by no means the correct explanation, let alone the intuition as to why you want to do this; it just gives you the flavor. You set the coefficients to zero because you don't have any better idea of what the numbers are going to be (and zero coefficients correspond nicely to no effect of the regressors, which is meaningful most of the time). On your second comment -- correct. $\endgroup$ – StasK Mar 4 '14 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.