Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to describe potential issues with neural language models as language generators, but think I'm missing something.

If I understand right, you can train a neural language model with cross-entropy loss against a vocabulary to properly model the probability of a sequence by decomposing it using Bayes Theorem like so:

P("The cat jumped") ~ P("jumped" | "The cat") * P("cat" | "The") * P("The")

So the globally optimal parameters for the model will accurately model P("The cat jumped").

If the training algorithm is some form of gradient descent, could you say that the way it approaches this is kind of greedy? As in, even though the a language model trains over the whole sequence, because the supervised examples only evaluate for single token continuation, the model is incentivized to correctly predict the next word more than subsequent words?

I feel as though my intuition is off here somehow, but not sure what I'm missing.


Edit: to add, this is a single-layer RNN, with left-to-right processing of a sequence, trained with cross-entropy loss for a single token continuation of ground truth.

My intuition is: 1) Perfect modeling of parameters (the global minima) will accurately model both: P(jumped | the cat) and P(cat | the ___ jumped) (I believe transformer and other models focus on this masked approach to better model long term context)

2) This particular left-to-right RNN trained with cross-entropy loss, due to the loss function and training objective, will bias toward a more accurate modeling of P(jumped | the cat) than of P(cat | the ___ jumped), if it has not yet found a global minima.


1 Answer 1


What you are describing is the most basic Markov Chain approach, which is kinda naive (in the technical sense).

You can only decompose the terms like that if each token is strictly dependent only on the previous token. The whole advancement had been brought about models that are higher-order.

For example, you could then decompose the probability of 'jumped' as:

$p('jumped'|'the\ cat') = p('jumped'|'the') + p('jumped'|'cat')p('cat'|'the')p('the')$

This is just an order-2 example since it's the maximum we can get with three words; it gets increasingly more baroque as you expand the context window.

  • 1
    $\begingroup$ Ah I see, thank you! For a left-to-right RNN approach, which I believe uses this basic decomposition, does this impact which local minima a model might be more inclined to? I.e. those that favor correct predictions of a word near in context versus one that is far. My understanding is that RNNs struggle with generation in long term context and I am trying to relate that the training objective (cross-entropy loss on next predicted token), but am not sure if I am making an accurate point. $\endgroup$ Dec 15, 2019 at 1:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.