Why does my neural net fail to learn higher frequency sine waves?

I am testing my neural network implementation. I have an input layer with a single unit, one hidden layer consisting of 65 tanh units, and an output layer consisting of a single linear output unit.

My data set consists of 100000 points $x_1, x_2, \ldots, x_{100000}$ sampled uniformly from $[-1,1]$, and the corresponding targets are $cos(16x_i)$, for $i=1, \ldots, 100000$.

I initialize the hidden layer's input weights to uniform random values in $[-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}]$ and the output layer's unit weights are uniform random values in $[-\frac{1}{\sqrt{66}}, \frac{1}{\sqrt{66}}]$.

I'm using a fixed learning rate of $\nu = 0.05$.

After training the network for $100 \times 100000$ steps, I don't seem to be getting good results. I test the network on $100000$ values in $[-1,1]$, and the resulting function looks nothing like the function $cos(16x)$.

I'm tracking the progress of the training by calculating the average error over 5000 steps, here's what it looks like:

When I use a simpler function such as $cos(4x)$ to train and then test, I get much better results, which leads me to believe that my implementation is OK for the most part.

Any suggestions on what might be going wrong? Did I need more hidden units and more training data? Do I need more layers? Should I be waiting longer for the gradient descent to converge? Some different learning rate? The above picture certainly looks like gradient descent has more or less converged, but those average errors still look a bit big?

• What kind of SGD are you using? Have you tried Adagrad or Adadelta? I would strongly suggest trying that before you add more layers etc. Subsequently, more layers means more non-linear transformations of your data, so I would try to see how it goes with 2 hidden layers and Adagrad. Finally, it is minor but how do you initialize your weights? A common approach is a random normal distribution multiplied with a scaling parameter of 0.05. Jun 14 '15 at 2:33
• I'm not sure which kind of SGD, so probably the most standard kind? I don't know what Adagrad and Adadelta, any good references for these? Weight initialization is described above. Jun 14 '15 at 3:10
• Try adagrad it's a safer choice, you can find some nice notes here ark.cs.cmu.edu/cdyer/adagrad.pdf. Which framework do you use for your implementation? Jun 14 '15 at 3:47
• @Fequish If you want to test your implementation, use gradient checking. Jun 14 '15 at 11:12
• @yasin.yazici What for of gradient checking do you mean? Is the my plot above basically gradient checking? Jun 14 '15 at 14:08

At first glance it looks like your neural net might be doing about as well as it can given the information it has. For a network with only a single input neuron, there is only so much you will be able to achieve in terms of creating a complicated output function. The output of your network is given by

$f(x) = b^{(2)} + w_1^{(2)}g(w_1^{(1)} x + b_1^{(1)}) + w_2^{(2)}g(w_2^{(1)} x + b_2^{(1)}) + ... + w_{65}^{(2)}g(w_{65}^{(1)} x + b_{65}^{(1)})$

where $g()$ is the $\tanh$ function in this case, $b$ is the bias, and $w_n$ is the weight mapping $x$ to the input. In other words, the output of your network will (optimally) be the best approximation achievable of your desired function with an expansion of 65 $\tanh$ functions. All your algorithm can do is find the weights that fit the function best, but there is no way for it to make a more complicated model.

Let's compare with a Taylor series of $\cos(x) = 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \frac{x^6}{6!} + ...$ just for a sense of the limitations here. If the argument of the cosine is small, you can get a good approximation by keeping just a few terms. As the argument of the cosine gets larger, you need more and more terms to get a reasonable approximation. So maybe 65 terms is enough to almost perfectly estimate $\cos(x)$ on the interval [-1, 1], does a pretty good job for $\cos(4x)$ (where the argument of cosine is on the interval [-4,4]), but fails completely for $\cos(16x)$ (cosine takes on values [-16,16]).

Since the above is just an educated guess as to your problems, here are some suggestions to check:

1. Restrict the interval to something like [-.1,.1] to see if you get a better approximation of $\cos(16x)$.

2. If this is indeed the problem your network is having, you can verify this is a high bias model by plotting your training error and your testing error. They should converge to about the same value for a high bias model.

3. If you find you do indeed have a high bias problem, the suggestion above to add another hidden layer or to increase the number of neurons in your hidden layer is a good one.

4. You can also think about adding additional features to your input layer if possible. One thought that comes to mind is adding a feature that is the argument of the cosine modulo $2\pi$ (i.e. $16x\mod 2\pi$ in this case).

• That's a 2 layer network, so it is a non-linear model of the input and the output of the network is $f(x)= b_1^{(2)} + w_1^{(2)}x *g(w_1^{(1)}x +b_1^{(1)})+ w_2^{(2)}x *g(w_2^{(1)}x +b_2^{(1)})+...+ w_{65}^{(2)}x *g(w_{65}^{(1)}x +b_{65}^{(1)})$. Also I don't aggree with representing higher frequencies with more units. $cos(x) = 1− \frac{x^2}{2}+...$ and $cos(4x) = 1− \frac{16x^2}{2}+...$ so, if the first one can be represented, the second one can also be represented just by multiplying by 16. My question is, how square of input can be represented by NN without multiplied by itself? Jun 15 '15 at 18:17
• @yasin.yazici I updated the equation in my answer to be what I think you meant, what you put doesn't look right because it couldn't pick up an $x$ in layer 2 (only the hidden layer "sees" $x$). Also you definitely can not just multiply by 16 to get the same Taylor series for $\cos(4x)$ as $\cos(x)$. The higher the frequency, the more terms you will need to approximate the function. My overall point was that there is a good chance you are not keeping enough terms in your expansion of $\cos(16x)$. You can test this with the suggestions I gave. Jun 15 '15 at 23:48
• I see my mistake on the equation. It should be $f(x)=b^{(2)}_1+w^{(2)}_1∗g(w^{(1)}_1x_1+b^{(1)}_1)+w^{(2)}_2∗g(w^{(1)}_2x_2+b^{(1)}_2)+...+w^{(2)}_{65}∗g(w^{(1)}_{65}x_{65}+b^{(1)}_{65})$. Why don't you use bias for each hidden units? I agree, now, why it is hard to approximate high frequencies as higher terms of Taylor expansion gets more prominent. I've tried a few network, cos(x) and cos(4x) is straight forward, but cos(8x) and higher terms couldn't be learnd even with 1000 of hidden units! Jun 16 '15 at 10:45
• Actually, more than one hidden layer can solve the issue. If the first layer can learn to convert input x into 8x, f_1(x) = 8x, then the remainer layers can solve f_2(x) = cos(x). When both of them combined f_2(f_1(x)) = cos(8x). I've tried a few network with 2 layers and the result are pretty good. But as the first layer in my explanation is linear it can be combined with the second layer and the overall network reduces to network with a single hidden layer. So the problem should be solved with even a single layer with resonable hidden units. What is the thing that I can't see? Jun 16 '15 at 11:45
• Good catch with the missing biases in the equation I wrote. It's interesting adding a second layer helped the network approximate the function better - do you have any reason to think the first layer is computing 8x and the second layer solving cos(x)? I would have guessed each layer would be doing something more complicated and harder to interpret than that. As for the difference between 2 layers or a single layer with more neurons I could only guess that $\tanh(\tanh(z))$ allows for a better approximation to a periodic function, but I'd have to think more about it for a while. Jun 16 '15 at 14:51