22
$\begingroup$

In a small data set ($n\sim100$ ) that I am working with, several variables give me perfect prediction/separation. I thus use Firth logistic regression to deal with the issue.

If I select the best model by AIC or BIC, should I include the Firth penalty term in the likelihood when computing these information criteria?

$\endgroup$
15
  • 2
    $\begingroup$ Would you mind explaining why it is unavoidable, since variable selection does not help with the "too many variables, too little sample size" problem? $\endgroup$ Mar 5, 2014 at 12:29
  • 4
    $\begingroup$ That is as bad as it gets. $\endgroup$ Mar 5, 2014 at 23:01
  • 1
    $\begingroup$ Have you considered treating this a Bayesian inference problem? Firth logistic regression is equivalent to MAP with jeffreys prior. You could use the fully laplace approximation to evalute marginal likelihoods - which is like an adjusted BIC (similar to AICc) $\endgroup$ Mar 9, 2014 at 1:38
  • 1
    $\begingroup$ @user, Because such variables usually predict only a handful of cases, and that is irreproducible: the true probability for that cell may be close to 90% say but with only two cases in it, you will get two ones 81% of the time. $\endgroup$
    – StasK
    Apr 4, 2015 at 12:53
  • 1
    $\begingroup$ Link to download K&K (1996) paper found on Google Scholar, bemlar.ism.ac.jp/zhuang/Refs/Refs/kitagawa1996biometrika.pdf $\endgroup$ May 30, 2015 at 0:40

1 Answer 1

3
$\begingroup$

If you want to justify the use of BIC: you can replace the maximum likelihood with the maximum a posteriori (MAP) estimate and the resulting 'BIC'-type criterion remains asymptotically valid (in the limit as the sample size $n \to \infty$). As mentioned by @probabilityislogic, Firth's logistic regression is equivalent to using a Jeffrey's prior (so what you obtain from your regression fit is the MAP).

The BIC is a pseudo-Bayesian criterion which is (roughly) derived using a Taylor series expansion of the marginal likelihood $$p_y(y) = \int L(\theta; y)\pi(\theta)\mathrm{d} \theta$$ around the maximum likelihood estimate $\hat{\theta}$. Thus it ignores the prior, but the effect of the latter vanishes as information concentrates in the likelihood.

As a side remark, Firth's regression also removes the first-order bias in exponential families.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.