45
$\begingroup$

What decides the choice of function ( Softmax vs Sigmoid ) in a Logistic classifier ?

Suppose there are 4 output classes . Each of the above function gives the probabilities of each class being the correct output . So which one to take for a classifier ?

$\endgroup$
  • 13
    $\begingroup$ The softmax function is nothing but a generalization of the sigmoid, so it's not entirely clear what you mean by "softmax vs. sigmoid." $\endgroup$ – dsaxton Sep 6 '16 at 15:56
  • 1
    $\begingroup$ As per what I understood , in softmax probabilties of all classes add to 1, but in sigmoid thats not the case. So when do we prefer sigmoid over softmax or vice versa ? $\endgroup$ – mach Sep 6 '16 at 16:02
  • 2
    $\begingroup$ It is the case with the sigmoid. When we use the sigmoid one class has probability $\exp(\beta^T x) / (\exp(\beta^T x) + 1)$ and the other has probability $1 / (\exp(\beta^T x) + 1)$. $\endgroup$ – dsaxton Sep 6 '16 at 16:05
  • 3
    $\begingroup$ The reddit poster is making a distinction that I think is wrong or at least irrelevant. Whether or not one of the classes has weight one is just a matter of shifting the scores, which has no effect on the probabilities. $\endgroup$ – dsaxton Sep 6 '16 at 19:03
  • 2
    $\begingroup$ Possible duplicate of Binary and multinomial logistic regression $\endgroup$ – Franck Dernoncourt Jan 2 '17 at 14:36
62
$\begingroup$

The sigmoid function is used for the two-class logistic regression, whereas the softmax function is used for the multiclass logistic regression (a.k.a. MaxEnt, multinomial logistic regression, softmax Regression, Maximum Entropy Classifier).


In the two-class logistic regression, the predicted probablies are as follows, using the sigmoid function:

$$ \begin{align} \Pr(Y_i=0) &= \frac{e^{-\boldsymbol\beta_ \cdot \mathbf{X}_i}} {1 +e^{-\boldsymbol\beta_0 \cdot \mathbf{X}_i}} \, \\ \Pr(Y_i=1) &= 1 - \Pr(Y_i=0) = \frac{1} {1 +e^{-\boldsymbol\beta_ \cdot \mathbf{X}_i}} \end{align} $$

In the multiclass logistic regression, with $K$ classes, the predicted probabilities are as follows, using the softmax function:

$$ \begin{align} \Pr(Y_i=k) &= \frac{e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} {~\sum_{0 \leq c \leq K}^{}{e^{\boldsymbol\beta_c \cdot \mathbf{X}_i}}} \, \\ \end{align} $$


One can observe that the softmax function is an extension of the sigmoid function to the multiclass case, as explained below. Let's look at the multiclass logistic regression, with $K=2$ classes:

$$ \begin{align} \Pr(Y_i=0) &= \frac{e^{\boldsymbol\beta_0 \cdot \mathbf{X}_i}} {~\sum_{0 \leq c \leq K}^{}{e^{\boldsymbol\beta_c \cdot \mathbf{X}_i}}} = \frac{e^{\boldsymbol\beta_0 \cdot \mathbf{X}_i}}{e^{\boldsymbol\beta_0 \cdot \mathbf{X}_i} + e^{\boldsymbol\beta_1 \cdot \mathbf{X}_i}} = \frac{e^{(\boldsymbol\beta_0 - \boldsymbol\beta_1) \cdot \mathbf{X}_i}}{e^{(\boldsymbol\beta_0 - \boldsymbol\beta_1) \cdot \mathbf{X}_i} + 1} = \frac{e^{-\boldsymbol\beta_ \cdot \mathbf{X}_i}} {1 +e^{-\boldsymbol\beta \cdot \mathbf{X}_i}} \\ \, \\ \Pr(Y_i=1) &= \frac{e^{\boldsymbol\beta_1 \cdot \mathbf{X}_i}} {~\sum_{0 \leq c \leq K}^{}{e^{\boldsymbol\beta_c \cdot \mathbf{X}_i}}} = \frac{e^{\boldsymbol\beta_1 \cdot \mathbf{X}_i}}{e^{\boldsymbol\beta_0 \cdot \mathbf{X}_i} + e^{\boldsymbol\beta_1 \cdot \mathbf{X}_i}} = \frac{1}{e^{(\boldsymbol\beta_0-\boldsymbol\beta_1) \cdot \mathbf{X}_i} + 1} = \frac{1} {1 +e^{-\boldsymbol\beta_ \cdot \mathbf{X}_i}} \, \\ \end{align} $$

with $\boldsymbol\beta = - (\boldsymbol\beta_0 - \boldsymbol\beta_1)$. We see that we obtain the same probabilities as in the two-class logistic regression using the sigmoid function. Wikipedia expands a bit more on that.

$\endgroup$
  • 1
    $\begingroup$ I am naive in this one, But I see this a lot of time β=−(β0−β1) What could be possible explanation to it? As far as I know in Sigmoids β would be a vector. And they are usually one for given run. Then how come β0 and β1 comes in the picture? $\endgroup$ – Ishan Bhatt Jan 18 '18 at 7:25
  • $\begingroup$ @IshanBhatt this comment may help. $\endgroup$ – Tom Hale May 4 '18 at 10:14
13
$\begingroup$

They are, in fact, equivalent, in the sense that one can be transformed into the other.

Suppose that your data is represented by a vector $\boldsymbol{x}$, of arbitrary dimension, and you built a binary classifier for it, using an affine transformation followed by a softmax:

\begin{equation} \begin{pmatrix} z_0 \\ z_1 \end{pmatrix} = \begin{pmatrix} \boldsymbol{w}_0^T \\ \boldsymbol{w}_1^T \end{pmatrix}\boldsymbol{x} + \begin{pmatrix} b_0 \\ b_1 \end{pmatrix}, \end{equation} \begin{equation} P(C_i | \boldsymbol{x}) = \text{softmax}(z_i)=\frac{e^{z_i}}{e^{z_0}+e^{z_1}}, \, \, i \in \{0,1\}. \end{equation}

Let's transform it into an equivalent binary classifier that uses a sigmoid instead of the softmax. First of all, we have to decide which is the probability that we want the sigmoid to output (which can be for class $C_0$ or $C_1$). This choice is absolutely arbitrary and so I choose class $C_0$. Then, my classifier will be of the form:

\begin{equation} z' = \boldsymbol{w}'^T \boldsymbol{x} + b', \end{equation} \begin{equation} P(C_0 | \boldsymbol{x}) = \sigma(z')=\frac{1}{1+e^{-z'}}, \end{equation} \begin{equation} P(C_1 | \boldsymbol{x}) = 1-\sigma(z'). \end{equation}

The classifiers are equivalent if the probabilities are the same, so we must impose:

\begin{equation} \sigma(z') = \text{softmax}(z_0) \end{equation}

Replacing $z_0$, $z_1$ and $z'$ by their expressions in terms of $\boldsymbol{w}_0,\boldsymbol{w}_1, \boldsymbol{w}', b_0, b_1, b'$ and $\boldsymbol{x}$ and doing some straightforward algebraic manipulation, you may verify that the equality above holds if and only if $\boldsymbol{w}'$ and $b'$ are given by:

\begin{equation} \boldsymbol{w}' = \boldsymbol{w}_0-\boldsymbol{w}_1, \end{equation} \begin{equation} b' = b_0-b_1. \end{equation}

$\endgroup$
  • $\begingroup$ @null Ok, I if you ask that, then you did not understand my explanation. Let me address your specific problem: if you tell me you are feeding your data to a sigmoid, then it must be a one-dimensional number, $x$. When feeding it to a sigmoid, you get the probability of $x$ being in one of your two classes, for instance $C_0$: $P(C_0|x)=σ(x)$. Then, the probability of $x$ being in $C_1$ is: $P(C_1|x)=1−P(C_0|x)=σ(x)$. Now let's replace your sigmoid by a softmax. (To be continued). $\endgroup$ – D... Jun 25 '17 at 23:49
  • $\begingroup$ (Continuation). In order to apply a softmax to a classification problem with two classes, you need your one dimensional data to be transformed into a two dimensional vector. Therefore, we need to define our $w_0$ and $w_1$. Let's choose $w_0=1$. Since $w_1$ must satisfy $w′=w_0−w_1$, we have $1=1−w_1$, so $w_1=0$. Now, we have $z_0=w_0x=x$ and $z_1=w_1x=0$. Using this, you can immediately verify that $σ(x)=\text{softmax}(z_0)$. $\endgroup$ – D... Jun 25 '17 at 23:49
  • $\begingroup$ Moreover, any combination of $w_0$ and $w_1$ that satisfies $w'=w_0-w_1$ (that is, $1=w_1-w_0$) would lead to the exact same result. This shows that the softmax has one redundant parameter. Although this may seem stupid, it is in fact an interesting property, since it allows normalization of the parameters $w_i$, which promotes numerical stability of the learning algorithm and inference. But this is just an extra comment, it is not important to answer your question :) $\endgroup$ – D... Jun 25 '17 at 23:50
  • $\begingroup$ Thanks a lot. I got it. In your first comment the probability $P(C_1|x)$ should probably be $1-\sigma(x)$. I now understand what is the idea behind the transformation. $\endgroup$ – null Jun 27 '17 at 11:05
  • $\begingroup$ Glad that you understood it ;) Yes, it's a typo, it obviously should be $P(C_1|x)=1 - \sigma(x)$. Thanks for pointing it out! $\endgroup$ – D... Jun 27 '17 at 11:19

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.