Many people add a symbol to the beginning of the decoder sentence for training, also known as the start sequence. For example,

Encoder input: "How are you today?"

Desired decoder input: "</s> I'm very good! </e>"

Why is this?


a seq2seq is typically implemented using an rnn. Mathematically, an rnn looks something like:

$$ h_{t} = f(h_{t-1}, i_t) \\ o_{t} = g(h_t) $$

... where:

  • $h_t$ is the hidden state at timestep $t$
  • $i_t$ is the input at timestep $t$
  • $o_t$ is the output at timestep $t$
  • $f( \cdot, \cdot)$ is some function, possibly linear, eg $\mathbf{W}\mathbf{x} + \mathbf{b}$, but could be more complicated than this
  • $g(\cdot)$ is another function, again could be linear

Then, seq2seq happens in two phases:

  • in phase 1, we take in the input sentence, one token at a one, and update the hidden state
  • in phase 2, we initialize the hidden state with the final hidden state from state 1, then predict outputs

For phase 2, looking at the equation, it is not enough to have just the hidden state to generate an output: we also need an input at each timestep.

What should be the input to the rnn at each timestep in phase 2?

  • for all but the first tokens, we can use the previous output
  • but what about for the very first output token
  • so we create a special 'start' token, which we feed in as the first token, when we are generating the output
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.