I am implementing a hierarchical sequence-to-sequence deep neural network model using long short-term memory (LSTM), where the bottom level of the hierarchy generates discrete outputs (characters from a set), but higher levels generate continuous vectors representing subsequences to be generated by lower levels. For the sake of this question, let's say I just have two levels in the hierarchy: characters (bottom level, generates discrete vectors representing character index) and words (top level, generates continuous latent vectors that are inputted to bottom layer LSTM to generate words as variable-length character sequences).

For the bottom level, I can just use a "stop" token as part of the character set to represent when to end the word. However, I also need to represent when to end the word sequence (sentence). There are many ways to represent a "stop" signal with continuous outputs (the outputs in the upper hierarchical levels), and I am wondering what the best approach to this is.

I've had some ideas:

  1. Use two stop signals in the bottom layer, one for words and one for sentences.
  2. A vector that represents a character sequence consisting of only the stop token is interpreted as a signal to stop generating the sentence.
  3. Use an additional "stop" neuron, with a value of 1 to stop generating the continuous sequence and 0 to continue. Loss calculation at the stop timestep would ignore the other output neurons.

Are any of these ideas preferable? If anyone else has approached a similar problem or can point me to a useful paper, please do enlighten me!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.