I'm writing my own implementation of a neural network to test my knowledge, and while it seems to run okay, it converges such that the output is always the mean value (0.5 since I'm using logistic output activation) regardless of the input, and nothing I do seems to change anything. There are a few similar questions on, but I've tried the suggestions there to no avail.
So far I've tried the network on an XOR gate, and on the function f(a + b) = c for a, b in the range [0, 0.5) using a squared difference loss function. Both of these networks have two inputs, one output, and I've tried a number of architectures, including 1 and 2 hidden layers with about 3-8 nodes in each. All of the neurons are using a logistic activation. Stochastic, mini-batch, with bias/without bias, all seemed to make no difference; same for the learning rate, for which I've done a number of values between 0 and 1. The only thing I can think of it that it's the initialisation (I'm currently initialising weights according to a uniform distribution with small min/max centred about 0) but wouldn't know how to fix that even if it were wrong.
I have looked at the deltas I'm getting for the weights and they seem sensible. After training for a while the weights look fairly sensible too, most between -1 and 1 though there are sometimes a few that are a lot larger. Training error starts off relatively high, then comes down within a few epochs and stays there, sometimes even increases slightly. Any help would be greatly appreciated.