My specific problem is to generate sequences of an algebraic style such as 1 + (3 * 4 + (1 + 5) * 6) etc.. I'm trying to use a long short term memoryLSTM algorithm for this task.

My problem is on how to train it.

The way I've been training these models so far is to have a batch with dimensions that describe (number of sequences, sequence length, number of features). So for instance, I would have for a sample batch 2 sequences, each sequence has 5 members in it, and each member consists of 1 feature.

I have been using lasagne so far, and it seems like the hidden and cell states are reset after each sequence (Stateless). So this brings me to my question.

If the hidden and cell states are reset and the dependencies within the sequence I am training on can vary in distance (i.e the number of members between two parenthesis), how does the LSTM capture these dependencies across sequences?

So for instance a sequence could have an opening parenthesis in it, and the following sequence it is closed. If an LSTM is cleared after every sequence, can it still capture this pattern? Would only a stateful LSTM be able to handle this task?

  • $\begingroup$ Also, as a followup, if the network is learning the connection between sequences in a batch, how are the hidden and cell states transferred to the network on the new sequence since the network just finished the previous one and would be at a position that is different than the position that the current sequence starts from. $\endgroup$ – terminix00 Nov 24 '16 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.