# Model selection with Firth logistic regression

In a small data set ($n\sim100$ ) that I am working with, several variables give me perfect prediction/separation. I thus use Firth logistic regression to deal with the issue.

If I select the best model by AIC or BIC, should I include the Firth penalty term in the likelihood when computing these information criteria?

• Would you mind explaining why it is unavoidable, since variable selection does not help with the "too many variables, too little sample size" problem? Mar 5, 2014 at 12:29
• That is as bad as it gets. Mar 5, 2014 at 23:01
• Have you considered treating this a Bayesian inference problem? Firth logistic regression is equivalent to MAP with jeffreys prior. You could use the fully laplace approximation to evalute marginal likelihoods - which is like an adjusted BIC (similar to AICc) Mar 9, 2014 at 1:38
• @user, Because such variables usually predict only a handful of cases, and that is irreproducible: the true probability for that cell may be close to 90% say but with only two cases in it, you will get two ones 81% of the time. Apr 4, 2015 at 12:53
If you want to justify the use of BIC: you can replace the maximum likelihood with the maximum a posteriori (MAP) estimate and the resulting 'BIC'-type criterion remains asymptotically valid (in the limit as the sample size $$n \to \infty$$). As mentioned by @probabilityislogic, Firth's logistic regression is equivalent to using a Jeffrey's prior (so what you obtain from your regression fit is the MAP).
The BIC is a pseudo-Bayesian criterion which is (roughly) derived using a Taylor series expansion of the marginal likelihood $$p_y(y) = \int L(\theta; y)\pi(\theta)\mathrm{d} \theta$$ around the maximum likelihood estimate $$\hat{\theta}$$. Thus it ignores the prior, but the effect of the latter vanishes as information concentrates in the likelihood.