My specific problem is to generate sequences of an algebraic style such as 1 + (3 * 4 + (1 + 5) * 6) etc.. I'm trying to use a long short term memoryLSTM algorithm for this task.
My problem is on how to train it.
The way I've been training these models so far is to have a batch with dimensions that describe (number of sequences, sequence length, number of features). So for instance, I would have for a sample batch 2 sequences, each sequence has 5 members in it, and each member consists of 1 feature.
I have been using lasagne so far, and it seems like the hidden and cell states are reset after each sequence (Stateless). So this brings me to my question.
If the hidden and cell states are reset and the dependencies within the sequence I am training on can vary in distance (i.e the number of members between two parenthesis), how does the LSTM capture these dependencies across sequences?
So for instance a sequence could have an opening parenthesis in it, and the following sequence it is closed. If an LSTM is cleared after every sequence, can it still capture this pattern? Would only a stateful LSTM be able to handle this task?