4
$\begingroup$

When vanilla neural networks are trained, we are simply trying to find a map from input to output, but with RNNs learning a sequence, we are trying to find a map from our input and a state vector to the output, and my question is then,

"How does an RNN ever learn the beginning of the sequence if its state vector is always random when learning the beginning or if it is polluted from the previous epoch of training on the end of some sequence?"

$\endgroup$

1 Answer 1

2
$\begingroup$

"How does an RNN ever learn the beginning of the sequence if its state vector is always random when learning the beginning or if it is polluted from the previous epoch of training on the end of some sequence?"

Typically, the hidden state of the first timestep (often denoted by $\mathbf{h}_0$) is the same for all sequences: as a result, there is no pollution from previous sequences or randomness. It is often chosen to be a zero vector.

FYI Learning initial state in RNNs

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.