I am currently implementing a simple neural network and the backprop algorithm in Python with numpy. I have already tested my backprop method using central differences and the resulting gradient is equal.
However, the network fails to approximate a simple sine curve. The network has one hidden layer (100 neurons) with $\tanh$ activation functions and a output layer with a linear activation function. Each unit also has a bias input. The training is done by simple gradient descent with a learning rate of 0.2.
The problem arises from the gradient, which gets larger with every epoch, but I don't know why. Further, the problem is unchanged if I decrease the learning rate.
EDIT: I have uploaded the code to pastebin: http://pastebin.com/R7tviZUJ