2
$\begingroup$

I don't entirely understand why logistic regression is based on an exponential function.

The sigmoid function seems to assume that as the dependent variables increase, the independent variable starts to increase exponentially. How do we know this is true? How do we know that this isn't a linear relationship?

$\endgroup$
7
  • 1
    $\begingroup$ stats.stackexchange.com/questions/88603/… $\endgroup$
    – Sycorax
    Aug 3 '17 at 1:37
  • 7
    $\begingroup$ Note that the model is for a probability $P(Y=1|\mathbf{X}=\mathbf{x})$ which must lie between 0 and 1. A linear function must eventually pass outside those limits, giving impossible probabilities. The logit is an example of a link function - by far the most popular - that stays within those limits. There are a variety of other functions that can be used (and indeed, both probit and complementary-log-log links are sometimes used instead of the logit link). $\endgroup$
    – Glen_b
    Aug 3 '17 at 2:06
  • $\begingroup$ @Glen_b nice concise explanation. I have to disagree with the logit being by far the most popular claim, though. In econometrics we usually resort to Probit link function. I think, that's because of the easier interpretability of the regression coefficients $\endgroup$
    – KenHBS
    Aug 3 '17 at 9:47
  • 2
    $\begingroup$ @Ken econometricians do use the probit a lot (and if I was talking only about the work done in econometrics you'd have a point), but econometricians are a small fraction of the people doing this kind of modelling. In applications across all areas I'd have guessed it would be over 90% logistic and less than 10% everything else put together (which would be almost all probit). [For questions here it's about 80/20 (logistic + logit to probit) but a somewhat higher proportion of new logistic questions would be closed as duplicates than would be the case for probit questions.] $\endgroup$
    – Glen_b
    Aug 3 '17 at 9:49
  • $\begingroup$ @Glen_b I understand that logit allows the function to stay within the 0-1 limits. However, that still doesn't answer my original question. How do we know that the independent variable increases exponentially, and why is it safe to assume that? $\endgroup$
    – bugsyb
    Aug 3 '17 at 23:26
6
$\begingroup$

[I didn't answer this before because I thought it would be a duplicate but I didn't locate a suitable one; I'll base a brief answer off my comments, at least until such a thread is located.]

Note that the model is for a probability $P(Y=1|\mathbf{X}=\mathbf{x})$ which must lie between 0 and 1.

A linear function must eventually pass outside those limits, giving impossible probabilities. That's usually undesirable.

The logit is an example of a link function - the most popular - that stays within those limits.

There are a variety of other functions that can be used for example, both probit and complementary-log-log links are sometimes used instead of the logit link for conditionally binomial models.

While convenient from several viewpoints, it's no more safe to assume it's specifically logistic than to assume it's anything else that satisfies the same restrictions, outside of the fact that it's often a reasonable approximation$-$as the famous saying goes, all models are wrong, but some are useful.

It makes about as much sense as to assume a relationship is linear in ordinary regression. It's convenient and often a good approximation, but typically it would be unwise to think the relationships are actually exactly linear.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.