0
$\begingroup$

I am implementing a hierarchical sequence-to-sequence deep neural network model using long short-term memory (LSTM), where the bottom level of the hierarchy generates discrete outputs (characters from a set), but higher levels generate continuous vectors representing subsequences to be generated by lower levels. For the sake of this question, let's say I just have two levels in the hierarchy: characters (bottom level, generates discrete vectors representing character index) and words (top level, generates continuous latent vectors that are inputted to bottom layer LSTM to generate words as variable-length character sequences).

For the bottom level, I can just use a "stop" token as part of the character set to represent when to end the word. However, I also need to represent when to end the word sequence (sentence). There are many ways to represent a "stop" signal with continuous outputs (the outputs in the upper hierarchical levels), and I am wondering what the best approach to this is.

I've had some ideas:

  1. Use two stop signals in the bottom layer, one for words and one for sentences.
  2. A vector that represents a character sequence consisting of only the stop token is interpreted as a signal to stop generating the sentence.
  3. Use an additional "stop" neuron, with a value of 1 to stop generating the continuous sequence and 0 to continue. Loss calculation at the stop timestep would ignore the other output neurons.

Are any of these ideas preferable? If anyone else has approached a similar problem or can point me to a useful paper, please do enlighten me!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.