Many people add a symbol to the beginning of the decoder sentence for training, also known as the start sequence. For example,

Encoder input: "How are you today?"

Desired decoder input: "</s> I'm very good! </e>"

Why is this?

  • $\begingroup$ I would add to your question if models are usually required to predict the SOS token too. $\endgroup$ Commented Jul 20, 2021 at 18:08

1 Answer 1


a seq2seq is typically implemented using an rnn. Mathematically, an rnn looks something like:

$$ h_{t} = f(h_{t-1}, i_t) \\ o_{t} = g(h_t) $$

... where:

  • $h_t$ is the hidden state at timestep $t$
  • $i_t$ is the input at timestep $t$
  • $o_t$ is the output at timestep $t$
  • $f( \cdot, \cdot)$ is some function, possibly linear, eg $\mathbf{W}\mathbf{x} + \mathbf{b}$, but could be more complicated than this
  • $g(\cdot)$ is another function, again could be linear

Then, seq2seq happens in two phases:

  • in phase 1, we take in the input sentence, one token at a one, and update the hidden state
  • in phase 2, we initialize the hidden state with the final hidden state from state 1, then predict outputs

For phase 2, looking at the equation, it is not enough to have just the hidden state to generate an output: we also need an input at each timestep.

What should be the input to the rnn at each timestep in phase 2?

  • for all but the first tokens, we can use the previous output
  • but what about for the very first output token
  • so we create a special 'start' token, which we feed in as the first token, when we are generating the output
  • $\begingroup$ is it common for the model to actually have to predict the SOS token? $\endgroup$ Commented Jul 20, 2021 at 18:07
  • 1
    $\begingroup$ @charlie, no it is not normal to have to predict the SOS token. The first token predicted is typically the first token after the SOS token. $\endgroup$ Commented Jul 26, 2021 at 2:05
  • $\begingroup$ ok thanks! Just for the sake of a sanity check/double checking. Predicting the EOS token is normal however, right? My guess is that without it it would be hard to know when to stop generating tokens during inference. So training to predict EOS seems important. $\endgroup$ Commented Jul 26, 2021 at 14:15
  • 1
    $\begingroup$ @charlie: yup. the network needs to learn when to stop. $\endgroup$ Commented Jul 27, 2021 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.