In Goodfellow et al.'s Deep Learning, the authors write about recurrent neural networks on page 371:

The total loss for a given sequence of $\mathbf{x}$ values paired with a sequence of $\mathbf{y}$ values would then be just the sum of the losses over all the time steps. For example, if $L^{(t)}$ is the negative log-likelihood of $y^{(t)}$ given $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}$, then $$\begin{align} &\phantom{=} L(\{\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(\tau)}\}, \{\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(\tau)}\}) \tag{10.12}\\ &=\sum_t L^{(t)}\tag{10.13}\\ &=-\sum_t \log p_{\mbox{model}}(y^{(t)} | \{\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}\}) \tag{10.14} \end{align}$$ where $p_{\mbox{model}}(y^{(t)} | \{\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}\})$ is given by reading the entry for $y^{(t)}$ from the model's output vector $\hat{\mathbf{y}}^{(t)}$.

Here, $\hat{\mathbf{y}}^{(t)} = \mbox{softmax}(\mathbf{o}^{(t)})$ and $\mathbf{o}^{(t)}$ is the output of the network for time $t$.


  1. Why is the likelihood of $y^{(t)}$ given $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(t)}$ first written as a joint likelihood over $\{\mathbf{x}, \mathbf{y}\}$ in equation (10.12)? I.e., if $\mathbf{x}$ is not being modeled how can you compute a likelihood for it? (This may be a more fundamental issue regarding my understanding of how a likelihood function relates to the probability of the data given the model.)
  2. In equation (10.14), why is there no sum over the length of $\hat{\mathbf{y}}$? Or is it just saying that you only need to consider the model's predicted probability for the true category of the observation? The latter doesn't make much sense to me because, intuitively, it feels like there should be a greater penalty if the uncertainty is distributed evenly among more categories compared to if it is distributed among just a few.
  3. For neural networks, how can you use a negative log probability as a surrogate for loss if noise is not being modeled?

1 Answer 1

  1. It's not explicitly stated that $L$ is a likelihood. $L^{(t)}$ is a likelihood, but $L$ is the loss which is just a function of the entire sequence $L(\{x^{(1)},...,x^{(\tau)}\},\{y^{(1)},...,y^{(\tau)}\})$. At each time step, the sequence of $x^{(1)},...,x^{(t)}$ is observed as shown in (10.14). It might be then that $L$ is the negative log likelihood because it's just a sum of negative log likelihoods, but then would't it be something like $$L(\{y^{(1)},...,y^{(\tau)}\}|\{x^{(1)},...,x^{(\tau)}\})=-\log p_{model}(\{y^{(1)},...,y^{(\tau)}\}|\{x^{(1)},...,x^{(\tau)}\})$$

  2. The label $y^{(t)}$ is a vector (as is $x^{(t)}$). For example with a binary classification problem $y^{(t)} \in \mathbb{R}^2$. The output $\hat{y} \in \mathbb{R}^2$ is also a vector of fixed length, but I believe they call it $o^{(t)}$.

  3. Could you rephrase/ make 3 more formal?

  • $\begingroup$ 1. So even though inputs $x^{(t)}$ are part of the loss function $L$, they don't play into the actual calculation because the loss function is the negative log likelihood, which assumes the $x^{(t)}$'s are given? 2. If $y^{(t)}$ is a vector, why is it not in boldface? Also, in their notation, the entry $y^{(t)}$ of $\hat{\mathbf{y}}^{(t)}$ is denoted without a hat. You are correct that the output is $\mathbf{o}^{(t)}$. $\endgroup$ Sep 9, 2017 at 16:30
  • $\begingroup$ 3. For a GLM, in which the noise is explicitly modeled, you could use the negative log likelihood as a loss function for gradient descent. In a neural network, the noise on the output is not being explicitly modeled, so are you implicitly making some assumptions about the noise distribution on the output by choosing to use a certain loss function? $\endgroup$ Sep 9, 2017 at 16:30
  • 1
    $\begingroup$ Just following up on my earlier comments. 1. I think this question is resolved thanks to your answer and my follow-up question in the above comment. 2. I think $y^{(t)}$ is not a vector but rather a label $\in \{1, 2, \ldots, K\}$, which is why it is not boldface. I was mistaken about their notation: $y^{(t)}$ is not an entry of $\hat{\mathbf{y}}^{(t)}$; it is the ground truth. 3. We do implicitly make assumptions about the noise on the output when fitting neural networks when we choose a loss function. $\endgroup$ Sep 12, 2017 at 18:50
  • $\begingroup$ Ah ok y(t) is an index $\endgroup$ Sep 12, 2017 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.