# How to choose between ordered logit and ordered probit regression?

If the dependent variable is discrete ordinal, like 0-10 then an ordered logit or ordered probit is appropriate to use. They are both similar but their interpretation are different and their error is different distributed. An ordered logit is logistically distributed and an ordered probit is normal distributed. The ordered logit have odds ratio while the ordered probit don't. Which method is used don't make a significance difference.

The above is what I read and understood from different sources. Even though I thought to plot the distribution of my data to see if it follows a logistic or normal distribution.

To do this I found that the plots below could be done. My question is then: is this a good way to choose between ordered logit or ordered probit?

library(effects)
library(fitdistrplus)

wvs.data = head(WVS) # data from the effects package
wvs.data
poverty religion degree country age gender
1       1      yes     no     USA  44   male
2       2      yes     no     USA  40 female
3       1      yes     no     USA  36 female
4       3      yes    yes     USA  25 female
5       1      yes    yes     USA  39   male
6       2      yes     no     USA  80 female

wvs.data$$poverty = as.numeric(wvs.data$$poverty)

fit1 <- fitdist(wvs.data$$poverty, "norm") fit2 <- fitdist(wvs.data$$poverty, "logis")

par(mfrow=c(2,2))
denscomp(list(fit1, fit2),legendtext = c("Normal", "Logistic"),
fitcol = c("green", "blue"))
qqcomp(list(fit1, fit2),legendtext = c("Normal", "Logistic"),
fitcol = c("green", "blue"))
cdfcomp(list(fit1, fit2),legendtext = c("Normal", "Logistic"),
fitcol = c("green", "blue"))
ppcomp(list(fit1, fit2),legendtext = c("Normal", "Logistic"),
fitcol = c("green", "blue"))


par(mfrow=c(1, 1))
descdist(wvs.data\$poverty, discrete = FALSE)
summary statistics
------
min:  1   max:  3
median:  1.5
mean:  1.666667
estimated sd:  0.8164966
estimated skewness:  0.8573214
estimated kurtosis:  2.7


• A good way to choose between a logit and a probit GLM is the Bayes factor, but there are other (non-Bayesian) options as well. Commented Aug 9 at 20:45
• There are 14 answers to what's essentially the same question on this page. In particular, this answer notes: "the empirical fit of the model to the data is unlikely to be of assistance in selecting a link, unless the shapes of the link functions in question differ substantially (of which, the logit and probit do not)."
– EdM
Commented Aug 9 at 21:04

It is not the outcome distribution that determines the best ordered choice model. What matters is the error term distribution in the underlying random utility or latent regression model (given the observed covariates). Unfortunately, this is not a testable assumption, and a compelling case for a particular distribution as best-in-class remains to be made. In short, looking at various characteristics of the outcome distribution while ignoring covariates is not helpful here.

The logit model has some mathematical features to recommend it, but any of these, such as computation of (effects on) odds ratios, can be replicated with the probit model under some assumptions and software-dependent inconvenience. Both models often give similar results: marginal effects, coefficients and cutpoints (except for scaling by ~1.7), z-tests, and predictions (unless the data is small or there are outliers). Both the ordered logit and probit require a parallel regression assumption, which is (sort of) testable and might lead you to a more complicated model that does alter the results.

Some criteria on how to choose between the two:

• the ease of computing what you care about: what is your research question/hypothesis, or how do you plan to use the model?
• the setting for your choice problem might favor one over the other: what's in your error term?
• do both to show that results do not hinge on the choice of ex-ante sensible models and stick one in the appendix

Your question lacks detail about these considerations, so it is hard to give more practical advice on a better approach.

• The two give similar fits so I always use the proportional odds model because the effects (odds ratios) are much easier to interpret than the estimated effects in a probit ordinal model. Logit also has some minor computational benefits. Commented Aug 9 at 20:47
• @FrankHarrell I find it much harder to think in terms of odds and prefer marginal effects on probabilities, which are the norm in my field. I have also found Ed Norton's critiques of ORs to be persuasive (e.g., doi.org/10.1111/1475-6773.14337 for binary outcomes). Is there a counterargument that you would recommend that I read here? Commented Aug 9 at 21:31
• That article and the HSR editorial policy are way, way off the mark IMHO. I’ve written multiple blog articles about this at fharrell.com . Marginal effects are highly problematic and don’t transport outside of the study. And if you think ORs are hard to interpret, just try to interpret a regression coefficient in a probit model. Note that it is a great idea to provide estimated differences in probabilities for selected covariate settings. That’s far different from marginal estimates using arbitrary covariate distributions though. Commented Aug 10 at 15:16
• Marginal effects are not what analysts think they are: fharrell.com/post/marg and odds ratios are likely to transport outside the study to the extent that either interactions are properly handled or the distribution of interacting factors is the same. Covariate-specific difference in risk are needed: fharrell.com/post/rdist Commented Aug 12 at 11:47
• The disparity in opinions comes from different definitions of marginal effects. What FrankHarrell calls "marginal effects" are odds ratios after removing most predictors and estimating another model with only one predictor for marginal relationship. What @dimitriy calls "marginal effects" are differences in probabilities when manipulating the treatment variable while fixing covariate settings at certain levels (usually at representative combinations, mean of each covariate, or over the observed distribution in the sample). The Health Services Research must have referred to the latter. Commented Aug 12 at 21:20

It is futile to "plot the distribution of my data to see if it follows a logistic or normal distribution."

• As Dimitriy implied, the distinction between the logit and probit models lies in the assumption of the latent, unobserved error term, not the characteristics of the observed level indicator.
• Unlike in the linear-model case, there is no residual in discrete models to estimate the error or test the error distribution. The observed outcome is a discrete event which is neither normally nor logistically distributed. In binary regression, the outcome indicator follows a Bernoulli distribution if each observation represents an individual, with a varying probability among the sample according to the specified model. If each observation represents a group, the probability of reaching an particular event count in a group follows a binomial distribution where the group size and event probability both vary among groups. In ordinal regression, the outcome indicator follows a multinomial distribution, with the event probability of each level varying among individuals and to be estimated. The fitted value, on the other hand, is an estimated event probability based on an assumption of the error-term distribution.
• Because an level indicator and the probability of a level measure totally different quantities, their difference does not represent an error term that is supposed to contrast two measurements of the same quantity. Residual quantities occurring in reference text on ordinal regression, such as Pearson and deviance residuals, are diagnostic and incomparable to residuals in linear models. See https://library.virginia.edu/data/articles/understanding-deviance-residuals.

The choice between logit and probit models is usually justified by theoretical reasons, computational convenience, and discipline convention.

• If the researcher has theoretical reasons for the error-term distribution, the choice can be argued accordingly. In a weight-loss study with the outcome measured as normal (BMI < 25), overweight (25 <= BMI < 30), or obese (BMI >= 30) for example, one may believe that the underlying BMI, which cause the ordinal categorization but is unavailable in the data, follows a normal distribution based on previous studies that have measured BMI directly. Therefore, the error term after accounting for predictor effects is also likely normally distributed, favoring the probit model. However, another can argue that the the latent error term may be logistically distributed even if the latent BMI is normally distributed, preferring a logit model. This is due to the distinction between marginal and conditional distribution. Regression assumptions are usually about the latter.
• Mathematically, logit and probit models of the same specification differ in their likelihood functions (i.e., probability mass functions) as shown at Difference between logit and probit models, just like how any type of model differs from another using the maximum likelihood estimator. The likelihood function emerges given the assumption of the distribution of the latent error term. Because the two assumed logistic and normal cumulative density functions (i.e. link functions) are extremely close in shape, as plotted at Difference between logit and probit models, logit and probit models of the same specification usually result in very similar goodness of fit, measured by log likelihoods, AIC, or BIC.
• However, it is not appropriate to overthrow one type over another solely based on goodness of fit, as their difference often happens by chance. If one must compare relative goodness of fit between a logit and a probit model for the same data, a perhaps acceptable approach is to record the differences in the log likelihoods among bootstrapped samples and to build a confidence interval or calculate a p value for the hypothesis that the logit and the probit model fit equally well. In this approach, measures like log likelihoods, AIC, or BIC are treated as individual observations. We take them over multiple samples to estimate the true parameter of goodness of fit.
• Economists seem to prefer probit models, which are deeply connected to linear models. When they need to account for random effects, instrument variables, and cross-equation correlation, an assumption of normally distributed error allows feasible model development out of a multivariate normal distribution. This is because a linear combination of normally-distributed random variables is still normally distributed, but a linear combination of logistically-distributed random variables is not logistically distributed. In engineering and health science, odds ratios prevail where authors usually present logit models, which are also faster to compute especially for multinomial outcomes.

More important than the choice between the two link functions is the assessment of linearity (i.e. location-effect specification), scale effects (i.e. variant error variance), and threshold-constant coefficients (i.e. parallel odds or parallel regression).

• These tests are all based on goodness-of-fit comparisons, giving a p value as the test result. Strategies afterwards are also stylish, as none of the tests is designed to identify the reason of violations. If the true model is y ~ x + I(x^2), whereas the specified model is y ~ x missing the squared term, test results will likely point to all of nonlinear, threshold-variant, and scale effects at the same time, suggesting different model specifications that may fit the data equally well. Selection among these specifications requires not a statistical process but subject expertise.
• Another phenomenon of which many may not be aware is that the choice between logit and probit link functions can sometimes lead to different location-effect specifications. For example, a specification of y ~ x may be sufficient in logit model, but a test of linearity could suggest y ~ x + I(x^2) when the link function is selected as probit. This usually happens only when the sample size is sufficiently large, with tens of thousands of observations to exploit the small difference in the tail distribution of logit and probit curves. The former suggests linear effects on the latent score, whereas the latter suggest parabolic effects. Likewise, one of logit and probit models may suggest scale or nominal effects while the other does not, due to fatter tails of the logistic distribution distinguishable from the normal ones in large samples.

Also more important than the choice between the two link functions are proper interpretations of the selected model.

• Assessing effect sizes solely on odds ratios can be problematic. Odds ratios can directly come from exponentiation of coefficients in logit models and be derived from predicted probabilities in probit models. Odds ratios estimated from different studies and groups are not directly comparable, unless one assumes that the unobserved error term has a constant variance across different studies and groups, which is not directly testable.
• Comparatively, comparisons based on marginal effects on probabilities are more reliable. This is typically done by calculating predicted probabilities at representative combinations of predictors, marginal effects at means, and average marginal effects across the observed predictor distribution. See the R package marginaleffects https://marginaleffects.com/files/marginaleffects_arel-bundock_greifer_heiss_jss5115.pdf for details and examples at https://marginaleffects.com/vignettes/logit.html on logit, at https://marginaleffects.com/vignettes/categorical.html on ordinal responses, and at https://marginaleffects.com/vignettes/hypothesis.html for average marginal effects. Both logit and probit models can give marginal effects on probabilities, which are easier to comprehend in common language than odds ratio. Although logit and probit models based on the same data may point to different mean structure, scale effects, or nominal effects, both models usually show very similar effects when translated onto category probabilities.
• When multiple models are plausible, model selection can introduce too much bias and subjectivity. Instead, a good practice is to present all results and discuss the implications from different perspectives. I recommend the R package ordinal to explore all these possibilities of scale and nominal effects with a variety of link functions.