My neural network can't even learn Euclidean distance

So I'm trying to teach myself neural networks (for regression applications, not classifying pictures of cats).

My first experiments were training a network to implement an FIR filter and a Discrete Fourier Transform (training on "before" and "after" signals), since those are both linear operations that can be implemented by a single layer with no activation function. Both worked fine.

So then I wanted to see if I could add an abs() and make it learn an amplitude spectrum. First I thought about how many nodes it would need in the hidden layer, and realized that 3 ReLUs are sufficient for a crude approximation of abs(x+jy) = sqrt(x² + y²), so I tested that operation by itself on lone complex numbers (2 inputs → 3 ReLU nodes hidden layer → 1 output). Occasionally it works: But most of the times that I try it, it gets stuck in a local minimum and fails to find the right shape:  I've tried all the optimizers and ReLU variants in Keras, but they don't make much difference. Is there something else I can do to make simple networks like this converge reliably? Or am I just approaching this with the wrong attitude, and you're supposed to just throw way more nodes than necessary at the problem and if half of them die it's not considered a big deal?

• Your plot shows what appears to be several different experiments, some of which work and some don't. What accounts for the differences? – Sycorax Aug 22 '18 at 17:04
• @Sycorax Those are just repeated trials of the same thing. Sometimes it finds a hexagonal pyramid solution, but usually it doesn't. – endolith Aug 22 '18 at 18:57
• Did you try with two hidden layers and a tanh as activation function? – Ketan Dec 2 '18 at 8:45
• @KetanNo, why? I posted a simpler variation here: stats.stackexchange.com/q/379884/11633 – endolith Dec 2 '18 at 15:12
• you might want to look at this one. stats.stackexchange.com/q/375655/27556. But can you explain why you think you only need 3 relus? the 'natural' decomposition would be one hidden layer to do the squaring approximation with relus and another layer to do the square root - basically relus are doing piecewise linear approximations. – seanv507 Dec 2 '18 at 16:01

I suspect that one possible cause of this is when $x+iy$ is skewed toward one coordinate, e.g. $x\gg y$, in which case you're trying to reproduce the identity, as then $abs(x+iy)\approx x$. There's really not much you can do here to remedy this. One option is to add more neurons as you've tried. The second option is to try a continuous activation, like a sigmoid, or perhaps something unbounded like an exponential. You could also try dropout (with say, 10% probability). You could use the regular dropout implementation in keras, which is hopefully smart enough to ignore situations when all 3 of your neurons drop out.