How do I make my neural network better at predicting sine waves?

Here, have a look: You can see exactly where the training data ends. Training data goes from $-1$ to $1$.

I used Keras and a 1-100-100-2 dense network with tanh activation. I calculate the result from two values, p and q as p / q. This way I can achive any size of number using only smaller than 1 values.

Please note I am still a beginner in this field, so go easy on me.

• To clarify, your training data is from around -1.5 to +1.5, so the network has learned that accurately? So your question is about extrapolating the result to unseen numbers outside the range of training data? – Neil Slater Oct 10 '17 at 14:46
• You could try Fourier transforming everything and working in the frequency domain. – Nick Alger Oct 11 '17 at 6:52
• To future reviewers: I don't know why this is being flagged for closure. It seems perfectly clear to me: it's about strategies for modeling periodic data with neural networks. – Sycorax Oct 11 '17 at 16:42
• I think it's a reasonable question for a beginner within the domain of machine learning, which we should accommodate here. I wouldn't close it – Aksakal Oct 11 '17 at 17:29
• I don't know if this will help, but out of the box a vanilla NN will only be able to learn a polynomial functions. In practice that is fine since you can make a polynomial arbitrarily close on a fixed interval. But it means that you can never learn a sine wave that extends on past the ends of the interval. The trick as other answers have pointed to below is to transform the problem into one that can be solved that way. That is what the Fourier transform suggested does, and in that case learning a sine wave is just learning a constant. – Ukko Oct 11 '17 at 21:34

You're using a feed-forward network; the other answers are correct that FFNNs are not great at extrapolation beyond the range of the training data.

However, since the data has a periodic quality, the problem may be amenable to modeling with an LSTM. LSTMs are a variety of neural network cell that operate on sequences, and have a "memory" about what they have "seen" before. The abstract of this book chapter suggests an LSTM approach is a qualified success on periodic problems.

In this case, the training data would be a sequence of tuples $(x_i, \sin(x_i))$, and the task to make accurate predictions for new inputs $x_{i+1} \dots x_{i+n}$ for some $n$ and $i$ indexes some increasing sequence. The length of each input sequence, the width of the interval which they cover, and their spacing, are up to you. Intuitively, I'd expect a regular grid covering 1 period to be a good place to start, with training sequences covering a wide range of values, rather than restricted to some interval.

(Jimenez-Guarneros, Magdiel and Gomez-Gil, Pilar and Fonseca-Delgado, Rigoberto and Ramirez-Cortes, Manuel and Alarcon-Aquino, Vicente, "Long-Term Prediction of a Sine Function Using a LSTM Neural Network", in Nature-Inspired Design of Hybrid Intelligent Systems)

• What is the sequence being modeled here? What are the time steps? This looks like a simple curve-fitting application to me. – David J. Harris Oct 10 '17 at 20:25
• @DavidJ.Harris I've updated my answer. – Sycorax Oct 10 '17 at 21:01
• what is the frequency of series is not $1/(2\pi)$? – Aksakal Oct 11 '17 at 11:37
• Is this how they do market predictions, for example? – Markus Appel Oct 11 '17 at 12:28
• No, that's not how you do market predictions. At least not how you do it to make money. – Aksakal Oct 11 '17 at 13:33

If what you want to do is learn simple periodic functions like this, then you could look into using Gaussian Processes. GPs allow you to enforce your domain knowledge to an extent by specifying an appropriate covariance function; in this example, since you know the data is periodic, you can choose a periodic kernel, then the model will extrapolate this structure.You can see an example in the picture; here, I'm trying to fit tide height data, so I know that it has a periodic structure. Because I'm using a periodic structure, the model extrapolates this periodicity (more or less) correctly. OFC if you're trying to learn about neural networks this isn't really relevant, but this might be a slightly nicer approach than hand-engineering features. Incidentally, neural networks and gp's are closely related in theory, so in principle there is some activation function you could choose that would do the same thing for a neural network

GPs aren't always useful because unlike neural nets, they are hard to scale to large datasets and deep networks, but if you're interested in low dimensional problems like this they will probably be faster and more reliable.

(in the picture, the black dots are training data and the red are the targets; you can see that even though it doesn't get it exactly right, the model learns the periodicity approximately. The coloured bands are the confidence intervals of the model's prediction)

• This plot is beautiful. – Sycorax Oct 10 '17 at 21:57

Machine learning algorithms - including neural networks - can learn to approximate arbitrary functions, but only in the interval where there is enough density of training data.

Statistics-based machine learning algorithms work best when they are performing interpolation - predicting values that are close to or in-between the training examples.

Outside of your training data, you are hoping for extrapolation. But there is no easy way to achieve that. A neural network never learns a function analytically, only approximately via statistics - this is true for nearly all supervised learning ML techniques. The more advanced algorithms can get arbitrarily close to a chosen function given enough examples (and free parameters in the model), but will still only do so in the range of supplied training data.

How the network (or other ML) behaves outside the range of your training data will depend on its architecture including the activation functions used.

The only way to have a machine learning algorithm predict a function analytically, is to build something into the assumptions of the model. For instance (and perhaps trivially), you could create features that equalled various $\sin$ functions of your input e.g $\text{sin}(x), \text{sin}(2x+\pi/4)$. The network - or even simpler, a linear regression - would learn to associate the most predictive value which is the closest $\sin$ function.

• "A neural network never learns a function analytically, only approximately via statistics." - the same can be said almost about anything that's used in practice, e.g. FFT doesn't learn analytically either. You could increase the sampling rate and period to infinity and get infinitely close to the true function, but the same is true about NN. – Aksakal Oct 10 '17 at 16:41
• @Aksakal: Yes that's true. However I didn't want to state that "All ML algorithms never learn functions analytically . . ." because someone no doubt would come up with a counter-example of some Bayesian analytical learner or genetic programming etc etc. I'll try to edit it to make it more general – Neil Slater Oct 10 '17 at 16:44

In some cases, @Neil Slater's suggested approach of transforming your features with a periodic function will work very well, and might be the best solution. The difficulty here is that you may need to choose the period/wavelength manually (see this question).

If you want the periodicity to be embedded more deeply into the network, the easiest way would be to use sin/cos as your activation function in one or more layers. This paper discusses potential difficulties and strategies for dealing with periodic activation functions.

Alternatively, this paper takes a different approach, where the weights of the network depend on a periodic function. The paper also suggests using splines instead of sin/cos, since they are more flexible. This was one of my favorite papers last year, so it's worth reading (or at least watching the video) even if you don't end up using its approach.

You took a wrong approach, nothing can be done with this approach to fix the issue.

There are several different ways to address the problem. I'll suggest the most obvious one through feature engineering. Instead of plugging time as a linear feature, put it as remainder of modulus T=1. For instance, t=0.2, 1.2 and 2.2 will all become a feature t1 = 0.1 etc. As long as T is larger than the period of wave, this will work out. Plug this thing into your net and see how it works.

Feature engineering is underrated. There's this trend in AI/ML where the sales men claim that you dump all your inputs into the net, and somehow it'll figure out what to do with them. Sure, it does, as you saw in your example, but then it breaks down as easily. This is a great example that show how important is to build good features even in some simplest cases.

Also, I hope you realize that this is the crudest example of feature engineering. It's just to give you an idea of what you could do with it.