6
$\begingroup$

I would like to intuitively understand the benefit of using the natural exponential in the sigmoid function used in logistic regression.

Why should it have to be $e^x$ instead of, for example $2^x$?

$\endgroup$
  • 5
    $\begingroup$ It is because the base of the log is e. If the base were 2 then you could use $2^x$. $\endgroup$ – Michael Chernick Apr 5 '18 at 7:00
  • 1
    $\begingroup$ To continue the Chernick comment, $2^x = e^{ln(2)x}$, so the choice of the base 2, e, 7.94, etc. is irrelevant. However, mathematically, the function $f(x) = e^x$ has the property the $f'(x) = f(x) $ , that is the derivative is itself, which makes a lot of computations easier. For example if $g(x) = 2^x$, then $g'(x) = ln(2)*g(x) $ and so computations are messier. In calculating the derivative of the sigmoid, the fact that $e^x$ is it's own derivative leads to a nice formula for the derivative of the sigmoid. $\endgroup$ – aginensky Apr 5 '18 at 15:16
  • 1
    $\begingroup$ The short answer is mathematical simplicity and convenience, as others explain. But an example with opposite flavour is that using a parameterisation in terms of $2^{-t/\tau}$ rather than of $\exp(-t/\tau)$ in modelling exponential decline with time $t$ (or distance) has one advantage in that $\tau$ is immediately a half-life or halving distance (with similar comments for growth and doubling). That can be a little more direct in reporting, especially to less numerate groups. $\endgroup$ – Nick Cox Apr 5 '18 at 15:25
  • $\begingroup$ Because a) any different logarithmic constant is just an ancillary parameter in an exponential family and b) a nice property of $\exp$ is that $\frac{\partial}{\partial x} \exp(x) = \exp(x)$. $\endgroup$ – AdamO Apr 5 '18 at 15:36
  • $\begingroup$ For logistic regression, bases like $2$ or $10$ aren't special and have nothing to recommend them. The exponential function is special because $e^x$ is approximately equal to $x$ for small $|x|$. This leads to simple interpretations of coefficients and is not true of any other base. (That's also why, historically, natural logarithms were the first ones invented and tabulated.) Thus, you should be asking the inverse of this question in circumstances where you do not see $e$ as the base of the exponential. $\endgroup$ – whuber Apr 5 '18 at 20:22
8
$\begingroup$

Because base $e$ is convenient, and it doesn't matter if you can freely scale your coefficient estimate.

Would using a functional form of $\frac{a^\mathbf{x\cdot b}}{1 + a^\mathbf{x\cdot b} }$ change your explanatory power? No.

Explanation:

I gave basically the same answer here for the softmax function.

Observe that $ e^ { \mathbf{x} \cdot \mathbf{b} \left( \ln a \right) } = a^ {\mathbf{x} \cdot \mathbf{b}}$. Hence:

$$ \frac{a^\mathbf{x\cdot b}}{1 + a^\mathbf{x\cdot b} } = \frac{e^\mathbf{x\cdot \tilde{b}}}{1 + e^\mathbf{x\cdot \tilde{b}} } $$

Where $\tilde{\mathbf{b}} = \left( \ln a \right) \mathbf{b} $. So using a different base than $e$ in the sigmoid function is the same as scaling your $\mathbf{b}$ vector.

$\endgroup$
  • 3
    $\begingroup$ It's nice not to have to carry factors of $\ln a$ when doing IRLS or backpropagation. $\endgroup$ – Bridgeburners Apr 5 '18 at 14:43
  • $\begingroup$ @Bridgeburners Could you elaborate? $\endgroup$ – Matthew Gunn Apr 5 '18 at 14:53
  • 2
    $\begingroup$ I just mean that, when doing optimization that requires taking the derivative of a loss function that includes sigmoids, if those sigmoids used base $a$ instead of base $e$ with its exponential expression, then we would have to track the factors of $\ln a$ that come from taking the derivative of those sigmoids. (Not that those two examples I listed always includes sigmoids.) $\endgroup$ – Bridgeburners Apr 5 '18 at 15:00
  • 2
    $\begingroup$ @Bridgeburners Got it. You're giving another reason why $\frac{d}{dx} e^x = e^x$ is a convenient property. $\endgroup$ – Matthew Gunn Apr 5 '18 at 15:10
7
$\begingroup$

In binary regression, one can use any cdf to relate the probability $\mathbb{P}(Y=1|\mathbf{x})$ and $\mathbf{x}$ in a generalised linear way $$\mathbb{P}(Y=1|\mathbf{x})=\Phi(\mathbf{x}^\text{T}\beta)$$as in

  1. logistic cdf, $\Phi(t)=1/\{1+1/e^t\}$
  2. probit (Normal) cdf, $\Phi(t)=\int_{-\infty}^t \varphi(x)\text{d}x$
  3. log-log cdf, $\Phi(t)=\exp\{-\exp(-x)\}$

The logistic offers some advantages, as making the conditional regression an exponential family model.

$\endgroup$
  • 1
    $\begingroup$ This is a good answer to a more general question: "Why sigmoid?" although OP asked "Why $e^x$ in the sigmoid?". It is implied that one need not take $e^x$ but could take, say, $x^2/(1-x^2)$ for a more locally linear activation function. Nonetheless, the three models you present are important to contextually understand modeling binary outcomes. Along those lines, I would point out that the main advantage of the sigmoid is that the linear model has coefficients interpreted as log-odds ratios. the complementary-log-log model estimates hazard ratio in a discrete time survival model. $\endgroup$ – AdamO Apr 6 '18 at 16:07
  • $\begingroup$ I meant the antiderivative of $(1-x^2)/(1+x^2)$ which is a scaled arctangent function $\endgroup$ – AdamO Apr 6 '18 at 16:16
5
$\begingroup$

For a Bernoulli likelihood, the variance is a function of the mean such that:

$$\text{var}(Y) = E(Y)(1-E(Y))$$

It turns out that a sigmoid function, also called the "inverse link" (for a logistic regression) function: $S(x) = \frac{\exp(x)}{1+\exp(x)}$ has the property that:

$$\frac{\partial}{\partial x} S(X) = S(X)(1-S(X))$$

It turns out this property holds for all GLMs using canonical parametrizations for exponential families.

$\endgroup$
  • 1
    $\begingroup$ Adam can you explain why this is important? $\endgroup$ – seanv507 Apr 5 '18 at 20:02
  • $\begingroup$ @seanv507 when we find the MLE for the canonical parameter, it achieves the Cramer-Rao lower bound. That's why logistic regression gives tighter confidence bounds for a sample proportion (transformed from the log-odds scale, e.g. $S^{-1}(p)$) than the normal approximation. $\endgroup$ – AdamO Apr 5 '18 at 20:32
  • $\begingroup$ @AdamO: actually, I wonder if in the case of the Bernoulli this exponential family requirement excludes any cdf $\Phi$. $\endgroup$ – Xi'an Apr 6 '18 at 9:39
  • $\begingroup$ @Xi'an Hmm. Well the RV with DF $\Phi$ is normally distributed which follows an exp family, agreeing with the known optimal conditions of linear regression. Or are you speaking of probit regression? $\endgroup$ – AdamO Apr 6 '18 at 16:02
  • 1
    $\begingroup$ @Xi'an . Or another way I might see your comment: is the probit less efficient than logistic regression because it is not ML for a Bernoulli outcome? In that case, you're right, logistic will have better coverage rates of $1-\alpha$ CIs. $\endgroup$ – AdamO Apr 6 '18 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.