I trained a lstm to forecast 3 different sin(x) with the frequencies of 0.5hz , 1hz and 2 hz.

During training and prediction: The lstm takes 25 time steps and outputs the next 50.

So the first layer has 25 inputs. The middle layer (hidden) has a LSTM cell with 150 units. The last layer has 50 outputs.

After the training the lstm can predict the 3 sin waves (0.5hz, 1hz, 2hz) pretty good. But it produces complete useless data when I want to forecast a sin with a frequency of 0.75hz or 4hz.

Because I'm a complete beginner with that I don't know if that is, because the LSTM may be overfitted or because the LSTM is not able to generalize the behaviour of this periodic function?

Maybe you can give me some tipps or a direction for further research :-)

Is it even possible to have a LSTM which can predict all kind (frequencies) of sins while just learning some specific frequencies?


My Code is based on the rnn.py from the sunsided/tensorflow-lstm-sin repo

The model is very similar to his third experiment


If I were to undertake this experiment, I would start with MAML. The basic idea is to train a NN so that the network can quickly adapt to a new, but similar, problem given a small amount of new data.

"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" by Chelsea Finn, Pieter Abbeel, Sergey Levine

We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model to be easy to fine-tune. We demonstrate that this approach leads to state-of-the-art performance on two few-shot image classification benchmarks, produces good results on few-shot regression, and accelerates fine-tuning for policy gradient reinforcement learning with neural network policies.

In Figure 2, the authors show that NNs trained using MAML can adapt to model sine waves of varying amplitude. I don't know if the same methods can be applied to sine waves of varying frequencies, but this seems like a promising avenue to explore.


In my personal experience (which is not that extensive), machine learning algorithms are usually better at interpolating than extrapolating. So, I am not surprised that your RNN fails for test frequencies higher than those found in the training set (4 Hz). However, it also seems to fail for 0.75 Hz...so, the only thing I can think of is to have more examples in the training set, for a larger varieties of frequency values: with that, the network will have a better chance of generalizing its behavior.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.