5
$\begingroup$

Concerning the choice of the link function in binomial regression (e.g. logit versus probit or cauchit), I wonder what the recommended comparison criterion might be. Note that I am not interested in hypothesis testing, but just in a relative criterion for "goodness".

Obvious candidates for comparing link functions are the AIC, the Brier score, or the test statistic of the Hosmer-Lemeshow test, but the latter depends on the somewhat arbitrary choice of binning quantiles.

Is one of these recommended, or are they even equivalent? Are there other suggestion?

I would suspect, e.g., that AIC and Brier score are asymptotically equivalent, because the Brier score is a measure for prediction accuracy and the AIC is asymptotically equivalent to using leave-one-out MSE as a criterion.

Improve this question
$\endgroup$
5
  • $\begingroup$ Related: stats.stackexchange.com/questions/20523/… and also this paper by Roger Koenker. Also: stats.stackexchange.com/questions/46833/…, stats.stackexchange.com/questions/10607/…, stats.stackexchange.com/questions/209659/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Dec 5, 2021 at 20:34
  • $\begingroup$ The link @kjetilbhalvorsen provides has some excellent context. One aspect not mentioned is logit is recommended if the sample size is small. I do not have a reference for this, but presumably this has to do with the higher kurtosis of the logistic compared to the normal distribution. $\endgroup$
    – Single Malt
    Commented Dec 5, 2021 at 20:56
  • 2
    $\begingroup$ @single-malt Are you referring to Hahn & Soyer (2005)? home.gwu.edu/~soyer/mv1h.pdf A comment in the thread mentioned by kjetil b halvorsen gives this reference: their finding was not that logistic was better for small sample size, but that probit was only better for medium and large sample sizes. Incidentally, this paper also gives a possible answer to my question: they used "DIC" (deviance information criterion). They used it because "s similar in interpretation and in spirit to [...] AIC". I thus guess that using the AIC is an acceptable appraoch. $\endgroup$
    – cdalitz
    Commented Dec 6, 2021 at 8:25
  • $\begingroup$ @kjetil-b-halvorsen The evaluation of parametrized link functions by Koenker (2007) is interesting. Unfortunately, the evaluation is done in Monte Carlo simulations with known true model and the distance to the true model is measured (criteria $d_1, d_2, d_\infty$ in Tbl. 4). These criteria cannot be computed from empirical data stemming from real observations. $\endgroup$
    – cdalitz
    Commented Dec 6, 2021 at 8:40
  • $\begingroup$ @cdalitz have been unable to substantiate the preference for logit over probit in small sample sizes. $\endgroup$
    – Single Malt
    Commented Dec 6, 2021 at 19:04

1 Answer 1

Reset to default
8
$\begingroup$

I think it should be a matter of interpretation of model results. The logit link has you modeling the log odds, or equivalently the multiplicative effect in the odds for a one-unit increase in a covariate. The probit link has you modeling the standard normal percentile, or equivalently, the additive effect in the standard normal percentile for a one-unit increase in a covariate.

From my experience as uncomfortable as odds and odds ratios can be they seem much more tangible and immediately relevant than standard normal percentiles unless the original endpoint was in fact a continuous normally distributed endpoint that was dichotomized for the purposes of binary regression. In that case the probit link would be relevant to talk in terms of percentiles or standard deviations from the mean. Likewise for the Cauchit link if the original data are in fact Cauchy distributed.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Although I find odds-ratios difficult to interpret, I agree that their algebraic (albeit non-linear) relation to the probabilities might be easier to deal with. This is an argument in favor of the logit link, if the results for different link functions are quite similar. If there is a noticeable difference, however, what is the criterion tells me which link function better represents the distribution in the data? $\endgroup$
    – cdalitz
    Commented Dec 6, 2021 at 7:50
  • $\begingroup$ I would use AIC because it is based on the likelihood and comes standard in most packages, though I suspect any differences in model fit will always be miniscule hence my recommendation based on interpretation. $\endgroup$
    – Geoffrey Johnson
    Commented Dec 6, 2021 at 14:41
  • 1
    $\begingroup$ As Geoffrey so well summarized, only the logit link provides an interpretable scale. I would also base the choice on which link function minimizes the likelihood ratio $\chi^2$ statistic for the chunk test of all two-way interactions combined. I.e., possibly choose the link that minimizes the need for interactions. $\endgroup$
    – Frank Harrell
    Commented Dec 6, 2021 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.