Background: I'm studying chapter 6 of Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville. In section 6.2.2.2 (pages 182 of 183 which can be viewed here) the use of sigmoid to output $P(y=1|x)$ is motivated.
To summarize some of the material they let $$z = w^Th+b$$ be an output neuron before an activation is applied where $h$ is the output of the previous hidden layer, $w$ is a vector of weights and $b$ is a scalar bias. The input vector is denoted $x$ (which $h$ is a function of) and the output value is denoted $y=\phi(z)$ where $\phi$ is the sigmoid function. The book wishes to define a probability distribution over $y$ using the value $z$. From the second paragraph of page 183:
We omit the dependence on $x$ for the moment to discuss how to define a probability distribution over $y$ using the value $z$. The sigmoid can be motivated by constructing an unnormalized probability distribution $\tilde P(y)$, which does not sum to 1. We can then divide by an appropriate constant to obtain a valid probability distribution. If we begin with the assumption that the unnormalized log probabilities are linear in $y$ and $z$, we can exponentiate to obtain the unnormalized probabilities. We then normalize to see that this yields a Bernoulli distribution controlled by a sigmoidal transformation of z: \begin{align} \log\tilde P(y) &= yz \\ \tilde P(y) &= \exp(yz) \\ P(y) &= \frac{\exp(yz)}{\sum_{y'=0}^1 \exp(y'z) } \\ P(y) &= \phi((2y-1)z) \end{align}
Questions: I'm confused about two things, particularly the first:
- Where is the initial assumption coming from? Why is the unnormalized log probability linear in $y$ and $z$? Can someone give me some inituition on how the authors started with $\log\tilde P(y) = yz$?
- How does the last line follow?