So I'm trying to teach myself neural networks (for regression applications, not classifying pictures of cats).

My first experiments were training a network to implement an FIR filter and a Discrete Fourier Transform (training on "before" and "after" signals), since those are both linear operations that can be implemented by a single layer with no activation function. Both worked fine.

So then I wanted to see if I could add an abs() and make it learn an amplitude spectrum. First I thought about how many nodes it would need in the hidden layer, and realized that 3 ReLUs are sufficient for a crude approximation of abs(x+jy) = sqrt(x² + y²), so I tested that operation by itself on lone complex numbers (2 inputs → 3 ReLU nodes hidden layer → 1 output). Occasionally it works:

3 ReLUs implementing Euclidean distance as inverted hexagonal pyramid

But most of the times that I try it, it gets stuck in a local minimum and fails to find the right shape:

3 ReLUs forming valley-shaped network

loss vs epochs

I've tried all the optimizers and ReLU variants in Keras, but they don't make much difference. Is there something else I can do to make simple networks like this converge reliably? Or am I just approaching this with the wrong attitude, and you're supposed to just throw way more nodes than necessary at the problem and if half of them die it's not considered a big deal?

  • 1
    $\begingroup$ Your plot shows what appears to be several different experiments, some of which work and some don't. What accounts for the differences? $\endgroup$ – Sycorax Aug 22 '18 at 17:04
  • $\begingroup$ @Sycorax Those are just repeated trials of the same thing. Sometimes it finds a hexagonal pyramid solution, but usually it doesn't. $\endgroup$ – endolith Aug 22 '18 at 18:57
  • $\begingroup$ Did you try with two hidden layers and a tanh as activation function? $\endgroup$ – Ketan Dec 2 '18 at 8:45
  • $\begingroup$ @KetanNo, why? I posted a simpler variation here: stats.stackexchange.com/q/379884/11633 $\endgroup$ – endolith Dec 2 '18 at 15:12
  • $\begingroup$ you might want to look at this one. stats.stackexchange.com/q/375655/27556. But can you explain why you think you only need 3 relus? the 'natural' decomposition would be one hidden layer to do the squaring approximation with relus and another layer to do the square root - basically relus are doing piecewise linear approximations. $\endgroup$ – seanv507 Dec 2 '18 at 16:01

The output seems to strongly suggest that one or more of your neurons goes dead (or perhaps the hyperplane of weights for two of your neurons have merged). You can see that with 3 Relu's, you get 3 shadowy splits in the center when you converge to the more reasonable solution. You can easily verify if this is true by checking the output values of each neuron to see if it stays dead for a large majority of your samples. Alternatively, you could plot all 2x3=6 neuron weights, grouped by their respective neuron, to see if two neurons collapse to the same pair of weights.

I suspect that one possible cause of this is when $x+iy$ is skewed toward one coordinate, e.g. $x\gg y$, in which case you're trying to reproduce the identity, as then $abs(x+iy)\approx x$. There's really not much you can do here to remedy this. One option is to add more neurons as you've tried. The second option is to try a continuous activation, like a sigmoid, or perhaps something unbounded like an exponential. You could also try dropout (with say, 10% probability). You could use the regular dropout implementation in keras, which is hopefully smart enough to ignore situations when all 3 of your neurons drop out.

  • 1
    $\begingroup$ +1, almost certainly this. When you use ReLU with such a small number of weights dead neurons almost always come into play. $\endgroup$ – kbrose Aug 22 '18 at 17:51
  • 1
    $\begingroup$ This is a plausible guess in the case of ReLU. But OP writes that they have tried a few different ReLU variants -- do dead neurons also occur in variants like ELU or Leaky Relu or PReLU? $\endgroup$ – Sycorax Aug 22 '18 at 18:01
  • $\begingroup$ 3 shadowy splits in the center when you converge to the more reasonable solution. Yes, that's the crude approximation I meant; an inverted hexagonal pyramid. or perhaps something unbounded like an exponential I did try elu and selu, which didn't work much better. two neurons collapse to the same pair of weights Ah, I hadn't thought of that; I just assumed they were dead. $\endgroup$ – endolith Aug 22 '18 at 18:55
  • $\begingroup$ I accessed the output of the hidden layer while using ReLU, and confirmed that usually one of the neurons is dead. Less often, two are overlapping and doing the same thing. I wonder if there's a way to make the neurons "repel" each other so they don't settle into redundant roles. $\endgroup$ – endolith Aug 23 '18 at 14:26
  • 1
    $\begingroup$ @endolith Dropout is explicitly designed around constructing independent neurons. ELUs never "die", but they do have vanishing gradient on the left. $\endgroup$ – Sycorax Aug 23 '18 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.