I am implementing a hierarchical sequence-to-sequence deep neural network model using long short-term memory (LSTM), where the bottom level of the hierarchy generates discrete outputs (characters from a set), but higher levels generate continuous vectors representing subsequences to be generated by lower levels. For the sake of this question, let's say I just have two levels in the hierarchy: characters (bottom level, generates discrete vectors representing character index) and words (top level, generates continuous latent vectors that are inputted to bottom layer LSTM to generate words as variable-length character sequences).
For the bottom level, I can just use a "stop" token as part of the character set to represent when to end the word. However, I also need to represent when to end the word sequence (sentence). There are many ways to represent a "stop" signal with continuous outputs (the outputs in the upper hierarchical levels), and I am wondering what the best approach to this is.
I've had some ideas:
- Use two stop signals in the bottom layer, one for words and one for sentences.
- A vector that represents a character sequence consisting of only the stop token is interpreted as a signal to stop generating the sentence.
- Use an additional "stop" neuron, with a value of 1 to stop generating the continuous sequence and 0 to continue. Loss calculation at the stop timestep would ignore the other output neurons.
Are any of these ideas preferable? If anyone else has approached a similar problem or can point me to a useful paper, please do enlighten me!