How do I make my neural network better at predicting sine waves?

Question

Here, have a look: You can see exactly where the training data ends. Training data goes from $-1$ to $1$.

I used Keras and a 1-100-100-2 dense network with tanh activation. I calculate the result from two values, p and q as p / q. This way I can achive any size of number using only smaller than 1 values.

Please note I am still a beginner in this field, so go easy on me.

Answer 1

You're using a feed-forward network; the other answers are correct that FFNNs are not great at extrapolation beyond the range of the training data.

However, since the data has a periodic quality, the problem may be amenable to modeling with an LSTM. LSTMs are a variety of neural network cell that operate on sequences, and have a "memory" about what they have "seen" before. The abstract of this book chapter suggests an LSTM approach is a qualified success on periodic problems.

In this case, the training data would be a sequence of tuples $(x_i, \sin(x_i))$, and the task to make accurate predictions for new inputs $x_{i+1} \dots x_{i+n}$ for some $n$ and $i$ indexes some increasing sequence. The length of each input sequence, the width of the interval which they cover, and their spacing, are up to you. Intuitively, I'd expect a regular grid covering 1 period to be a good place to start, with training sequences covering a wide range of values, rather than restricted to some interval.

(Jimenez-Guarneros, Magdiel and Gomez-Gil, Pilar and Fonseca-Delgado, Rigoberto and Ramirez-Cortes, Manuel and Alarcon-Aquino, Vicente, "Long-Term Prediction of a Sine Function Using a LSTM Neural Network", in Nature-Inspired Design of Hybrid Intelligent Systems)

Answer 2

If what you want to do is learn simple periodic functions like this, then you could look into using Gaussian Processes. GPs allow you to enforce your domain knowledge to an extent by specifying an appropriate covariance function; in this example, since you know the data is periodic, you can choose a periodic kernel, then the model will extrapolate this structure.You can see an example in the picture; here, I'm trying to fit tide height data, so I know that it has a periodic structure. Because I'm using a periodic structure, the model extrapolates this periodicity (more or less) correctly. OFC if you're trying to learn about neural networks this isn't really relevant, but this might be a slightly nicer approach than hand-engineering features. Incidentally, neural networks and gp's are closely related in theory, so in principle there is some activation function you could choose that would do the same thing for a neural network

GPs aren't always useful because unlike neural nets, they are hard to scale to large datasets and deep networks, but if you're interested in low dimensional problems like this they will probably be faster and more reliable.

(in the picture, the black dots are training data and the red are the targets; you can see that even though it doesn't get it exactly right, the model learns the periodicity approximately. The coloured bands are the confidence intervals of the model's prediction)

Answer 3

Machine learning algorithms - including neural networks - can learn to approximate arbitrary functions, but only in the interval where there is enough density of training data.

Statistics-based machine learning algorithms work best when they are performing interpolation - predicting values that are close to or in-between the training examples.

Outside of your training data, you are hoping for extrapolation. But there is no easy way to achieve that. A neural network never learns a function analytically, only approximately via statistics - this is true for nearly all supervised learning ML techniques. The more advanced algorithms can get arbitrarily close to a chosen function given enough examples (and free parameters in the model), but will still only do so in the range of supplied training data.

How the network (or other ML) behaves outside the range of your training data will depend on its architecture including the activation functions used.

The only way to have a machine learning algorithm predict a function analytically, is to build something into the assumptions of the model. For instance (and perhaps trivially), you could create features that equalled various $\sin$ functions of your input e.g $\text{sin}(x), \text{sin}(2x+\pi/4)$. The network - or even simpler, a linear regression - would learn to associate the most predictive value which is the closest $\sin$ function.

Answer 4

In some cases, @Neil Slater's suggested approach of transforming your features with a periodic function will work very well, and might be the best solution. The difficulty here is that you may need to choose the period/wavelength manually (see this question).

If you want the periodicity to be embedded more deeply into the network, the easiest way would be to use sin/cos as your activation function in one or more layers. This paper discusses potential difficulties and strategies for dealing with periodic activation functions.

Alternatively, this paper takes a different approach, where the weights of the network depend on a periodic function. The paper also suggests using splines instead of sin/cos, since they are more flexible. This was one of my favorite papers last year, so it's worth reading (or at least watching the video) even if you don't end up using its approach.

Answer 5

You took a wrong approach, nothing can be done with this approach to fix the issue.

There are several different ways to address the problem. I'll suggest the most obvious one through feature engineering. Instead of plugging time as a linear feature, put it as remainder of modulus T=1. For instance, t=0.2, 1.2 and 2.2 will all become a feature t1 = 0.1 etc. As long as T is larger than the period of wave, this will work out. Plug this thing into your net and see how it works.

Feature engineering is underrated. There's this trend in AI/ML where the sales men claim that you dump all your inputs into the net, and somehow it'll figure out what to do with them. Sure, it does, as you saw in your example, but then it breaks down as easily. This is a great example that show how important is to build good features even in some simplest cases.

Also, I hope you realize that this is the crudest example of feature engineering. It's just to give you an idea of what you could do with it.

Answer 6

You can train the neural network on the autoregressive principle, i.e. based on N previous values. The value of the argument is not required. Forecasting is done in the same way based on previous values, including those predicted. This works fine:

from sklearn.neural_network import MLPRegressor
import numpy as np

step = 0.1
X = np.arange(0, 10, step)
real_sin = np.sin(X)
X_train,y_train = [],[]
start = 10 # training history
for i in range(0,len(real_sin)-start):
    X_train.append(real_sin[i:i+start])
    y_train.append(real_sin[i+start])

regr = MLPRegressor(max_iter=1000, hidden_layer_sizes= tuple([100]*10)).fit(X_train, y_train)

X_new,Y_new = [X[-1]],[real_sin[-1]]
X_in = y_train[-start:]
for i in range(200):
    X_new.append(X_new[-1]+step)
    next_y =  regr.predict([X_in])[0]
    Y_new.append(next_y)
    X_in.append(next_y)
    X_in.pop(0)

The result should look like this: