# Conditional probability vs. likelihood - neural networks

In Goodfellow et al.'s Deep Learning, the authors write about recurrent neural networks on page 371:

The total loss for a given sequence of $\mathbf{x}$ values paired with a sequence of $\mathbf{y}$ values would then be just the sum of the losses over all the time steps. For example, if $L^{(t)}$ is the negative log-likelihood of $y^{(t)}$ given $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}$, then \begin{align} &\phantom{=} L(\{\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(\tau)}\}, \{\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(\tau)}\}) \tag{10.12}\\ &=\sum_t L^{(t)}\tag{10.13}\\ &=-\sum_t \log p_{\mbox{model}}(y^{(t)} | \{\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}\}) \tag{10.14} \end{align} where $p_{\mbox{model}}(y^{(t)} | \{\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}\})$ is given by reading the entry for $y^{(t)}$ from the model's output vector $\hat{\mathbf{y}}^{(t)}$.

Here, $\hat{\mathbf{y}}^{(t)} = \mbox{softmax}(\mathbf{o}^{(t)})$ and $\mathbf{o}^{(t)}$ is the output of the network for time $t$.

Questions:

1. Why is the likelihood of $y^{(t)}$ given $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}$ first written as a joint likelihood over $\{\mathbf{x}, \mathbf{y}\}$ in equation (10.12)? I.e., if $\mathbf{x}$ is not being modeled how can you compute a likelihood for it? (This may be a more fundamental issue regarding my understanding of how a likelihood function relates to the probability of the data given the model.)
2. In equation (10.14), why is there no sum over the length of $\hat{\mathbf{y}}$? Or is it just saying that you only need to consider the model's predicted probability for the true category of the observation? The latter doesn't make much sense to me because, intuitively, it feels like there should be a greater penalty if the uncertainty is distributed evenly among more categories compared to if it is distributed among just a few.
3. For neural networks, how can you use a negative log probability as a surrogate for loss if noise is not being modeled?

1. It's not explicitly stated that $L$ is a likelihood. $L^{(t)}$ is a likelihood, but $L$ is the loss which is just a function of the entire sequence $L(\{x^{(1)},...,x^{(\tau)}\},\{y^{(1)},...,y^{(\tau)}\})$. At each time step, the sequence of $x^{(1)},...,x^{(t)}$ is observed as shown in (10.14). It might be then that $L$ is the negative log likelihood because it's just a sum of negative log likelihoods, but then would't it be something like $$L(\{y^{(1)},...,y^{(\tau)}\}|\{x^{(1)},...,x^{(\tau)}\})=-\log p_{model}(\{y^{(1)},...,y^{(\tau)}\}|\{x^{(1)},...,x^{(\tau)}\})$$
2. The label $y^{(t)}$ is a vector (as is $x^{(t)}$). For example with a binary classification problem $y^{(t)} \in \mathbb{R}^2$. The output $\hat{y} \in \mathbb{R}^2$ is also a vector of fixed length, but I believe they call it $o^{(t)}$.
• 1. So even though inputs $x^{(t)}$ are part of the loss function $L$, they don't play into the actual calculation because the loss function is the negative log likelihood, which assumes the $x^{(t)}$'s are given? 2. If $y^{(t)}$ is a vector, why is it not in boldface? Also, in their notation, the entry $y^{(t)}$ of $\hat{\mathbf{y}}^{(t)}$ is denoted without a hat. You are correct that the output is $\mathbf{o}^{(t)}$. Commented Sep 9, 2017 at 16:30
• Just following up on my earlier comments. 1. I think this question is resolved thanks to your answer and my follow-up question in the above comment. 2. I think $y^{(t)}$ is not a vector but rather a label $\in \{1, 2, \ldots, K\}$, which is why it is not boldface. I was mistaken about their notation: $y^{(t)}$ is not an entry of $\hat{\mathbf{y}}^{(t)}$; it is the ground truth. 3. We do implicitly make assumptions about the noise on the output when fitting neural networks when we choose a loss function. Commented Sep 12, 2017 at 18:50