Motivating sigmoid output units in neural networks starting with unnormalized log probabilities linear in $z=w^Th+b$ and $\phi(z)$

Question

Background: I'm studying chapter 6 of Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville. In section 6.2.2.2 (pages 182 of 183 which can be viewed here) the use of sigmoid to output $P(y=1|x)$ is motivated.

To summarize some of the material they let $$z = w^Th+b$$ be an output neuron before an activation is applied where $h$ is the output of the previous hidden layer, $w$ is a vector of weights and $b$ is a scalar bias. The input vector is denoted $x$ (which $h$ is a function of) and the output value is denoted $y=\phi(z)$ where $\phi$ is the sigmoid function. The book wishes to define a probability distribution over $y$ using the value $z$. From the second paragraph of page 183:

We omit the dependence on $x$ for the moment to discuss how to define a probability distribution over $y$ using the value $z$. The sigmoid can be motivated by constructing an unnormalized probability distribution $\tilde P(y)$, which does not sum to 1. We can then divide by an appropriate constant to obtain a valid probability distribution. If we begin with the assumption that the unnormalized log probabilities are linear in $y$ and $z$, we can exponentiate to obtain the unnormalized probabilities. We then normalize to see that this yields a Bernoulli distribution controlled by a sigmoidal transformation of z: \begin{align} \log\tilde P(y) &= yz \\ \tilde P(y) &= \exp(yz) \\ P(y) &= \frac{\exp(yz)}{\sum_{y'=0}^1 \exp(y'z) } \\ P(y) &= \phi((2y-1)z) \end{align}

Questions: I'm confused about two things, particularly the first:

1. Where is the initial assumption coming from? Why is the unnormalized log probability linear in $y$ and $z$? Can someone give me some inituition on how the authors started with $\log\tilde P(y) = yz$?
2. How does the last line follow?

Answer 1

There are two possible outcomes for $y \in \{0, 1\}$. It's very important, because this property changes meaning of the multiplication. There are two possible cases:

\begin{align} \log\tilde P(y=1) &= z \\ \log\tilde P(y=0) &= 0 \\ \end{align}

In addition important to notice that unnormalized logarithmic probability for $y=0$ is constant. This property derives from the main assumption. Applying any deterministic function to the constant value will produce constant output. This property will simplify final formula when we will do normalization over all possible probabilities, because we just need to know only unnormalized probability for $y=1$ and for $y=0$ it's always constant. And since output from the network in unnormalized logarithmic probability we will require only one output, because another one assumed to be constant.

Next, we are applying exponentiation to the unnormalized logarithm probability in order to obtain unnormalized probability.

\begin{align} \tilde P(y=1) &= e ^ z \\ \tilde P(y=0) &= e ^ 0 = 1 \end{align}

Next we just normalize probabilities dividing each unnormalized probability by the sum of all possible unnormalized probabilities.

\begin{align} P(y=1) = \frac{e ^ z}{1 + e ^ z} \\ P(y=0) = \frac{1}{1 + e ^ z} \end{align}

We are interested only in $P(y=1)$, because that's what probability from the sigmoid function means. The obtained function doesn't look like sigmoid on the first look, but they are equal and it's easy to show.

\begin{align} P(y=1) = \frac{e ^ x}{1 + e ^ x} = \frac{1}{\frac{e ^ x + 1}{e ^ x}} = \frac{1}{1 + \frac{1}{e ^ x}} = \frac{1}{1 + e ^ {-x}} \end{align}

The last statement can be confusing at first, but it just a way to show that that final probability function is a sigmoid. The $(2y−1)$ value converts $0$ to $-1$ and $1$ to $1$ (or we can say that it would be without change).

$$ P(y) = \sigma((2y - 1)z) = \begin{cases} \sigma(z) = \frac{1}{1 + e ^ {-z}} = \frac{e ^ z}{1 + e ^ z} & \text{when } y = 1 \\ \sigma(-z) = \frac{1}{1 + e ^ {-(-z)}} = \frac{1}{1 + e ^ z} & \text{when } y = 0 \\ \end{cases} $$

As we can see, it just the way to show the relation between $\sigma$ and $P(y)$

Answer 2

I also find this fragment of the book challenging to follow, and the above answer by itdxer deserves quite some time to understand as well for someone who's not properly fluent with probabilities and maths thinking. I made it however by reading the answer backwards, so start with the sigmoid of z

\begin{align} P(y=1) = \frac{e ^ z}{1 + e ^ z} = \frac{1}{1 + e ^ {-z}} \end{align}

and try to follow back to.

\begin{align} \log\tilde P(y) &= yz \end{align}

Then it makes sense why they started the explanation with yz - it's by design, same as the final

\begin{align} \sigma((2y-1)z) \end{align}

by construction allows to get -1 for y=0 and 1 for y=1, which are the only possible values of y under the Bernoulli.

Answer 3

Here's a more formal phrasing that will appeal to those with a measure-theoretic background.

Let $Y$ be a Bernoulli r.v. and let $P_Y$ denote the pushforward measure, i.e for $y\in \{0,1\}$, $P_Y(y)=P(Y=y)$ and let $\tilde P_Y$ denote its unnormalized counterpart.

We have the following chain of implications:

$$\begin{aligned} \log \tilde P_Y(y)=yz &\implies \tilde P_Y(y) = \exp(yz)\\ &\implies P_Y(y) = \frac{e^{yz}}{e^{0\cdot z}+e^{1\cdot z}}=\frac{e^{yz}}{1+e^{ z}}\\ &\implies P_Y(y) =y\frac{e^{z}}{1+e^{ z}} + (1-y)\frac{1}{1+e^{ z}}\\ &\implies P_Y(y) =y\sigma(z) + (1-y)\sigma(-z)\\ &\implies P_Y(y) = \sigma((2y-1)z) \end{aligned}$$

The last equality is a smart way of mapping $\{0,1\}$ to $\{-1,1\}$

Answer 4

I will try to expand on the answer of @itdxer. From the comment section of the answer, it seems that the doubt centres around the justification of the line $log(\tilde{P}(y))=yz$, which becomes $z$ for $y=1$ and $0$ for $y=0$. What might be the justification for taking such a form? I will try to provide some insight into that.

At $y=1$, as per the above formula $-log(\tilde{P}(y))=-z$.

At $y=0$, $-log(\tilde{P}(y))=0$.

We will consider both the cases where $y=1$ and $y=0$.

Case I - Original value of $y$ is 1

If we take negative-log as the cost function and concentrate on $\tilde{P}(y=1)$, gradient descent on the cost function has the tendency to push $z$ towards the right side. This is what we want.

However, if originally $y=0$, we require $z$ to be pushed towards the left side. The negative-log cost being $0$, fails to deliver that.

Thus the above formula $log(\tilde{P}(y))=yz$ along with negative-log cost has the capability to learn cases with $y=1$ but fails to do the same for $y=0$ cases.

Case II - Original value of $y$ is 0

We could instead have started off with the formula

$-log(\tilde{P}(y))=(1-y)z$

$-log(\tilde{P}(y))=z$ for $y=0$

$-log(\tilde{P}(y))=0$ for $y=1$

Here the negative-log cost of $\tilde{P}(y=0)$ has the capability to push $z$ towards the left side, which is what we want when $y=0$. But, the negative-log cost of $\tilde{P}(y=1)$, being 0, fails to push $z$ towards the right.

Therefore, both formulas can deliver on one set of $y$ values but fails on the other. The final formula for $-log(\tilde P(y))$ should be such that it can select the preferred scenario based on the original value of $y$.

I am providing below a plot, which will clarify the point.

Thus we see that for cases with $y=1$, $\sigma(z)$ is the choice for the output unit; for $y=0$ cases, $\sigma(-z)$ is the preferred choice for output unit.

$\sigma((2y-1)z)$ happens to be the unified formula that can make this interchange possible based on the value of $y$. Besides, we can see that if we start with either of the formulas

$-log(\tilde{P}(y))=-yz$ or

$-log(\tilde{P}(y))=(1-y)z$

we can end up with $P(y)=\sigma((2y-1)z)$, progressing as done in the book or in the answer of @itdxer. Although not proved here, I think that this happy ending happens due to the identity $\sigma(-z)=1-\sigma(z)$.

Finally, I would like to mention that, personally, I feel motivating the equation $P(y)=\sigma((2y-1)z)$ would have been more appealing by first showing the plots I gave above. Then explaining that each of $\sigma(z)$ or $\sigma(-z)$ would be suitable for only one type of cases. Hence, we require some transformation capable of making this switch based on the value of $y$.