Sequence to Sequence models: Why is the start sequence (e.g. "") required? Many people add a symbol to the beginning of the decoder sentence for training, also known as the start sequence. For example, 
Encoder input: "How are you today?"

Desired decoder input: "</s> I'm very good! </e>"

Why is this? 
 A: a seq2seq is typically implemented using an rnn. Mathematically, an rnn looks something like:
$$
h_{t} = f(h_{t-1}, i_t) \\
o_{t} = g(h_t)
$$
... where:


*

*$h_t$ is the hidden state at timestep $t$

*$i_t$ is the input at timestep $t$

*$o_t$ is the output at timestep $t$

*$f( \cdot, \cdot)$ is some function, possibly linear, eg  $\mathbf{W}\mathbf{x} + \mathbf{b}$, but could be more complicated than this

*$g(\cdot)$ is another function, again could be linear


Then, seq2seq happens in two phases:


*

*in phase 1, we take in the input sentence, one token at a one, and update the hidden state

*in phase 2, we initialize the hidden state with the final hidden state from state 1, then predict outputs


For phase 2, looking at the equation, it is not enough to have just the hidden state to generate an output: we also need an input at each timestep.
What should be the input to the rnn at each timestep in phase 2?


*

*for all but the first tokens, we can use the previous output

*but what about for the very first output token

*so we create a special 'start' token, which we feed in as the first token, when we are generating the output

