Do recurrent neural language models greedily model language probability?

Question

Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to describe potential issues with neural language models as language generators, but think I'm missing something.

If I understand right, you can train a neural language model with cross-entropy loss against a vocabulary to properly model the probability of a sequence by decomposing it using Bayes Theorem like so:

P("The cat jumped") ~ P("jumped" | "The cat") * P("cat" | "The") * P("The")

So the globally optimal parameters for the model will accurately model P("The cat jumped").

If the training algorithm is some form of gradient descent, could you say that the way it approaches this is kind of greedy? As in, even though the a language model trains over the whole sequence, because the supervised examples only evaluate for single token continuation, the model is incentivized to correctly predict the next word more than subsequent words?

I feel as though my intuition is off here somehow, but not sure what I'm missing.

Thanks!

Edit: to add, this is a single-layer RNN, with left-to-right processing of a sequence, trained with cross-entropy loss for a single token continuation of ground truth.

My intuition is: 1) Perfect modeling of parameters (the global minima) will accurately model both: P(jumped | the cat) and P(cat | the ___ jumped) (I believe transformer and other models focus on this masked approach to better model long term context)

2) This particular left-to-right RNN trained with cross-entropy loss, due to the loss function and training objective, will bias toward a more accurate modeling of P(jumped | the cat) than of P(cat | the ___ jumped), if it has not yet found a global minima.

Answer 1

What you are describing is the most basic Markov Chain approach, which is kinda naive (in the technical sense).

You can only decompose the terms like that if each token is strictly dependent only on the previous token. The whole advancement had been brought about models that are higher-order.

For example, you could then decompose the probability of 'jumped' as:

$p('jumped'|'the\ cat') = p('jumped'|'the') + p('jumped'|'cat')p('cat'|'the')p('the')$

This is just an order-2 example since it's the maximum we can get with three words; it gets increasingly more baroque as you expand the context window.