15
$\begingroup$

I've been thinking about this problem. The usual logistic function for modeling binary data is: $$ \log\left(\frac{p}{1-p}\right)=\beta_0+\beta_1X_1+\beta_2X_2+\ldots $$ However is the logit function, which is an S-shaped curve, always the best for modeling the data? Maybe you have reason to believe your data does not follow the normal S-shaped curve but a different type of curve with domain $(0,1)$.

Is there any research into this? Maybe you can model it as a probit function or something similar, but what if it is something else entirely? Could this lead to better estimation of the effects? Just a thought I had, and I wonder if there is any research into this.

|cite|improve this question
$\endgroup$
  • 3
    $\begingroup$ possible duplicate of Difference between logit and probit models $\endgroup$ – Macro Jan 19 '13 at 5:38
  • 2
    $\begingroup$ @macro I don't think it's an exact duplicate. That question is about just logit and probit; this one asks for other alternatives as well. $\endgroup$ – Peter Flom♦ Jan 19 '13 at 13:01
  • $\begingroup$ I'm voting to leave this open. The main difference I see is that this Q is asking for research in statistics on the topic of different possible link functions. It's a subtle difference, but it may be enough. @Glen, you may want to review the other Q, if you haven't seen it already. In my answer I talk about different possible links. If you think this Q isn't really different, flag it & the mods can close it; if you can think of a way to make the distinction b/t what you're asking & that Q clearer, you may want to edit to do so. $\endgroup$ – gung♦ Jan 19 '13 at 15:12
  • $\begingroup$ I know it's not an exact duplicate of the logit vs. probit question but I thought gung's answer, which went above and beyond what was asked by the linked question, addresses most of what was asked here, which is why I closed as a duplicate. There are probably other closely related threads but that is the first that came to mind. $\endgroup$ – Macro Jan 19 '13 at 20:50
  • $\begingroup$ Thanks for the comments. I believe my question is different from the previous question. I am very familiar with the probit and log-log transformations, and the discussion from the previous question was very informative for me. However, I am interested in other link functions (possibly non-parametric?) that are possible, in a situation you which you may or may not have knowledge that the probability curve follows a different distribution. I think when interactions are involved among the covariates this could play an important role. @David J. Harris answer is helpful as well... $\endgroup$ – Glen Jan 20 '13 at 5:01
15
$\begingroup$

People use all sorts of functions to keep their data between 0 and 1. The log-odds fall out naturally from the math when you derive the model (it's called the "canonical link function"), but you're absolutely free to experiment with other alternatives.

As Macro alluded to in his comment on your question, one common choice is a probit model, which uses the quantile function of a Gaussian instead of the logistic function. I've also heard good things about using the quantile function of a Student's $t$ distribution, although I've never tried it.

They all have the same basic S-shape, but they differ in how quickly they saturate at each end. Probit models approach 0 and 1 very quickly, which can be dangerous if the probabilities tend to be less extreme. $t$-based models can go either way, depending on how many degrees of freedom the $t$ distribution has. Andrew Gelman says (in a mostly unrelated context) that $t_7$ is roughly like the logistic curve. Lowering the degrees of freedom gives you fatter tails and a broader range of intermediate values in your regression. When the degrees of freedom go to infinity, you're back to the probit model.

Hope this helps.

Edited to add: The discussion @Macro linked to is really excellent. I'd highly recommend reading through it if you're interested in more detail.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ The question is specifically about "binary data" — not about data that is between 0 and 1. The probit model has no theoretical justification in the case of binary data. $\endgroup$ – Neil G Jan 19 '13 at 16:03
  • 3
    $\begingroup$ @NeilG, one reason to use the probit model is that it gives a convenient way of modeling multivariate binary data (e.g. with a mixed model) as thresholded normals. In that case, the correlation matrix of the underlying variables is statistically indentifiable, whereas it is not in the logistic case. There's a bit longer discussion here. $\endgroup$ – Macro Jan 19 '13 at 20:45
  • $\begingroup$ @Macro: Oh, I see. That's very interesting, thanks. $\endgroup$ – Neil G Jan 19 '13 at 20:52
  • $\begingroup$ @David J.Harris: Do you mean quintile (or maybe quantile has the same meaning), i.e., breaking down the distribution into chunks of fifths: 20%, 40%,..,100% ? $\endgroup$ – MSIS Aug 26 '16 at 17:20
  • 1
    $\begingroup$ @MSIS a quintile divides into fifths, a percentile divides into 100ths, and a quartile divides into arbitrary units See en.wikipedia.org/wiki/Quantile#Specialized_quantiles $\endgroup$ – David J. Harris Aug 26 '16 at 19:43
11
$\begingroup$

I see no reason, a-priori, why the appropriate link function for a given dataset has to be the logit (although the universe does seem to be rather kind to us in general). I don't know if these are quite what you're looking for, but here are some papers that discuss more exotic link functions:

Disclosure: I don't know this material well. I tried dabbling with the Cauchit and Scobit a couple of years ago, but my code kept crashing (probably because I'm not a great programmer), and it didn't seem relevant for the project I was working on, so I dropped it.

Most of this stuff has to do with differing tail behavior than the prototypical links (i.e., the function 'turns the corner' early and doesn't asymptote to 0 & 1 very fast), or are skewed (i.e., like the cloglog, they approach one limit faster than the other). You should also be able to replicate these behaviors, I believe, by fitting a spline function of $X$ with a logistic link.

|cite|improve this answer
$\endgroup$
4
$\begingroup$

The best strategy is to model the data in light of what is going on (No surprise!)

  • Probit models originate with LD50 studies - you want the dose of insecticide that kills half the bugs. The binary response is whether the bug lives or dies (at a given dose). The bugs that are susceptible at one dose will be susceptible at lower doses as well, which is where the idea of modeling to the cumulative Normal comes in.
  • If the binary observations come in clusters, you can use a beta-binomial model. Ben Bolker has a good introduction in the documentation of his bbmle package (in R) which implements this in simple cases. These models allow more control over the variation of the data than what you get in a binomial distribution.
  • Multivariate binary data -- the sort that rolls up into multi-dimensional contingency tables - can be analysed using a log-linear model. The link function is the log rather than the log odds. Some people refer to this as Poisson regression.

There probably isn't research on these models as such, although there has been plenty of research on any one of these models, and on the comparisons between them, and on different ways of estimating them. What you find in the literature is that there is a lot of activity for a while, as researchers consider a number of options for a particular class of problems, and then one method emerges as superior.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ +1 for beta-binomial. That's a great tool to have in one's toolbox. $\endgroup$ – David J. Harris Jan 21 '13 at 4:20
3
$\begingroup$

Logit is a model such that the inputs are a product of experts each of which is a Bernoulli distribution. In other words, if you consider all of the inputs to be independent Bernoulli distributions with probabilities $p_i$ whose evidence is combined, you will find that you are adding the logistic function applied to each of the $p_i$s. (Another way of saying the same thing is that conversion from the expectation parametrization to the natural parametrization of the Bernoulli distribution is the logistic function.)

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.