5
$\begingroup$

I am trying to find sigmoid function alternatives for logistic regression. I am curious that if I can replace sigmoid function by any cumulative distribution function, and what will be the best?

$\endgroup$
  • 1
    $\begingroup$ The reason why it is called logistic regression is because the inverse function of sigmoid function is logit-function. $\endgroup$ – Lerner Zhang Jan 29 '17 at 6:37
  • $\begingroup$ Why cumulative distribution function? $\endgroup$ – Lerner Zhang Jan 29 '17 at 12:09
  • $\begingroup$ As Lerner indicates, the particular sigmoid function used by logistic regression is what makes it logistic. If you change that, it would still be a binomial GLM but would no longer be logistic regression. Commonly GLM packages offer several alternatives for binomial regression models, such as probit models and complementary log-log models. However, "best" depends on what you're trying to optimize (as well as on other things). $\endgroup$ – Glen_b Jan 29 '17 at 12:23
  • $\begingroup$ @Lerner because cdfs have the properties you tend to want from such a function. $\endgroup$ – Glen_b Jan 29 '17 at 12:34
  • 1
    $\begingroup$ @Lerner I'm not at all sure what you're asking in the second half of that. In relation to the first half, the relevant properties of a cdf are $F$ is monotonic nondecreasing with $0\leq F\leq 1$; these would both be absolute minimal requirements for such a model, though we'd usually want monotonic increasing (for which we just need a continuous cdf which is nonzero within its domain), and probably an easily calculated derivative (which is usually a matter of picking a distribution with a convenient density). $\endgroup$ – Glen_b Jan 30 '17 at 9:32
1
$\begingroup$

Not all distributions' CDFs are sigmoid. Consider the CDF of the uniform distribution (figure copied from Wikipedia):

enter image description here

Other distributions may be less obvious, but still problematic. Consider the CDF of a Gamma distribution with $k=.5,\ \theta=1$ (the lavender line at the far left; figure copied from Wikipedia):

enter image description here

At a minimum, you are going to need a distribution whose support is $(-\infty, \infty)$ before you could consider using its CDF as a link function.

There are many (I suppose infinite) possible link functions that can be used, though. You don't have to use the logit, and it isn't necessarily the best (although we need to be more precise about what "best" means). You may be interested in reading my answers here: Difference between logit and probit models, or here: Is the logit function always the best for regression modeling of binary data?

$\endgroup$
  • $\begingroup$ Thanks for the answer and links @gung. I was aware of your answer on the second link and have found many sigmoid like functions. But I have some other questions arising naturally like that i) Is it appropriate to call resulting new models as logistic regression? ii) How will the interpretations of estimates? We have the old logit and we learned to interpret it. $\endgroup$ – Sadik Candan Apr 27 '16 at 12:39
  • $\begingroup$ I recognise now that some parts of my above questions are answered on those two links. But even I might need still some elaboration. $\endgroup$ – Sadik Candan Apr 27 '16 at 12:47
0
$\begingroup$

I believe tanh(z) is a good replacement for the sigmoid function. Behaves almost exactly the same way. Also, the gradient descent update expression is the same

$\endgroup$
  • $\begingroup$ Yes tanh is an alternative. What about any cumulative distribution function? Can they be used? $\endgroup$ – Sadik Candan Mar 29 '16 at 11:12
  • $\begingroup$ Graphically, the cdf of a normal distribution looks the same as the sigmoid. So I think that would be a good alternative. $\endgroup$ – Agrima Bahl Mar 29 '16 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.