Simulating data with long range dependencies

Question

I want to evaluate how well a recurrent neural network I've created captures long-range dependencies, and the effects altering the network have on this.

I'm not entirely sure how I would go about doing this except by training it on data where I already have an understanding of the underlying dependencies. As such, I'd like a way to simulate data where I can control the underlying long range dependencies of the data.

To be honest, I'm quite new to deep learning so I'm still getting my head around what I'm actually doing. However, I've read that "long memory" can be defined in terms of autocorrelations; that is, that the autocorrelations decay to zero very slowly. My idea was to generate sequences that have this property, train the network on it and then let the network come up with sequences of it's own, which one would hope would retain this property.

I just don't know how I would go about generating such data!

Answer 1

You could use a toy problem where it is impossible to do well without learning long range dependencies. For example, in the parity problem the model would read in a series of 0's and 1's and be trained to classify sequences as having an even or an odd number of 1's.