Feedforward neural network for sinusoidal prediction

Question

This is 100% curiosity, so I apologize if the question is under-constrained. If so, please comment with where my thinking is misguided!

Can a feedforward neural network predict a sinusoidal relationship? In the simplest case, where y can be reliably predicted as sin(x) (that is, f(x) = sin(x)). Can this be accomplished without recurrent nodes?

Apologies if the maths are obvious! Without an LSTM or other augmentation, can a standard neural network learn a sinusoidal relationship from back-propagation?

I'm most curious about theoretical constraints. Are there provably lower bounds on the number of nodes or layers that would be required to model the relationship (if possible)?

As a toy example, I put together a really, really simple example with the scikit MLP regressor:

from sklearn.neural_network import MLPRegressor

import matplotlib.pyplot as plt
import numpy as np

Fs = 8000
f = 5
sample = 8000
x = np.arange(sample)

# CHOOSE Y 
y = np.sin(2 * np.pi * f * x / Fs) # sine
y = x.copy() # linear sanity check of network performance

model = MLPRegressor()

num_sims = 100

with_noise = np.empty((num_sims, sample))

plt.figure()
for i in range(num_sims):
    with_noise[i, :] = np.random.rand(1, sample) + y
    plt.plot(with_noise[i,:])

plt.show()

# now with a neural network
x_expanded = np.tile(x, (num_sims, 1))
model.fit(x_expanded, with_noise)

predictions = model.predict(x.reshape(1, -1))

plt.figure()
plt.plot(x, np.squeeze(predictions))
plt.show()

In the linear example (i.e., the y = x.copy(), I'm paranoid about shallow copies...), the network, unsurprisingly, does a fine job. In the case of the sine, it falls down pretty hard.

Again, just a curiosity! I don't typically work with periodic data, so this was interesting to me.

Answer 1

Let's tackle the issue of RNNs first.

It depends what you mean by learning the sine wave. If you mean just learning $\sin(x)$ given $x$, then you don't need RNNs. RNNs are used to predict sequences, and your problem is not a sequence problem - you want to predict a mapping $f: \mathbb{R} \to [0, 1]$, and not a sequence where the value at next time step depends on previous timesteps.

If on the other hand you want to learn sine wave as a time series (I mean to be able to predict $\sin(x+\delta)$ given previous examples, then it is a valid approach (but I think this is a really poor, as it is an exotic example, better ones could be considering problems for which Hidden Markov Models/Linear Dynamical Systems are used, for example predicting behavior of a system where $y_t = Ax_t + b$ for some matrix $A$).

I'm most curious about theoretical constraints. Are there provably lower bounds on the number of nodes or layers that would be required to model the relationship (if possible)?

Yes, the universal approximation theorem says that it is possible if you restrict to learning on a bounded subset of real numbers (even under the constraint of one hidden layer).

Note that unfortunately, this theorem doesn't give practical estimates for a number of nodes to achieve $\epsilon$-approximation (although this might be extracted, at least in some cases from proofs - from what I remember they are essentially based on Fourier transforms and/or Stone-Weierstrass theorem).