I have coded a neural network with 1 hidden layer and 1 numerical output. No biases.
By appropiate choice of the activation function, I can easily approximate some continuous function.
However, even something as simple as
f(x) = 50 if x >= 100, and 25 otherwise
I cannot approximate using my neural net. I've tried changing learning rate, and also number of hidden notes, but it simply won't converge to it.
Why does it do so poorly at this incredibly simple function? What can I do to make it converge? It is not just this function, but an other similarly non-continuous function.