# How does SGD come in the picture for Sequence to Sequence models?

I was learning that seq2seq models (from the deeplearning.ai course) try to maximize:

$$\max_{y} P_{\theta}(y_1 \dots y_{T'} \mid x_1 \dots x_T )$$

I learned that one way they do it is via beam search. However, what I don't understand is where SGD comes in the picture. I will explain the way I think it comes in the picture and the true objective I think we are trying to optimize. I think we are actually trying to optimize:

$$\max_{\theta,y} P_{\theta}(y_1 \dots y_{T'} \mid x_1 \dots x_T ) = \max_{\theta,y} J(\theta,y)$$

we want to optimize the most likely sequence but we also want to improve the RNN parameters (or at least that's what I would have thought, specially since in normal supervised learning thats what we do).

So according to that I believe this is the way it would be done:

• at iteration $$t$$ we have model $$\theta^{(t)}$$
• we get it's most likely sequence with beam search $$\hat y = y_1 \dots y_{T'} := BeamSearch(x_1...x_T,\theta^{(t)})$$
• the using that most likely sequence we do SGD on the parameters according to the (normalized log likelihood) $$\theta^{(t+1)} := \theta^{(t)} - \eta_t J(\theta^{(t)},\hat y)$$ which basically makes the current most likely sequence even more likely with SGD

• repeat

I think this is probably wrong because it would seem this algorithm reinforces a specific sentence. If the first guess wasn't right I'd assume it would be some sort of horrible confirmation bias algorithm...but I can't see how SGD comes in the picture according to the descriptions I've read in Coursera's deeplearning.ai!

A sequence to sequence model produces a probability distribution over sequences $$Y$$ conditioned on another sequence $$X$$.
At test time, you may want to find the mode of that distribution: $$\max_Y P(Y|X;\theta)$$ -- the sequence which is most likely. For example if you have a translation model, you want to find the translation which has the highest probability under the model. This maximization must be done with beam search.
At training time, you already have $$X,Y$$ pairs from the data and you are simply trying to maximize the (log) likelihood of the model: $$\max_\theta E[\log P(Y|X;\theta)]$$. This is a completely different task from test time, and doesn't require beam search, and is done with SGD.