10
$\begingroup$

So I'm trying to teach myself neural networks (for regression applications, not classifying pictures of cats).

My first experiments were training a network to implement an FIR filter and a Discrete Fourier Transform (training on "before" and "after" signals), since those are both linear operations that can be implemented by a single layer with no activation function. Both worked fine.

So then I wanted to see if I could add an abs() and make it learn an amplitude spectrum. First I thought about how many nodes it would need in the hidden layer, and realized that 3 ReLUs are sufficient for a crude approximation of abs(x+jy) = sqrt(x² + y²), so I tested that operation by itself on lone complex numbers (2 inputs → 3 ReLU nodes hidden layer → 1 output). Occasionally it works:

3 ReLUs implementing Euclidean distance as inverted hexagonal pyramid

But most of the times that I try it, it gets stuck in a local minimum and fails to find the right shape:

3 ReLUs forming valley-shaped network

loss vs epochs

I've tried all the optimizers and ReLU variants in Keras, but they don't make much difference. Is there something else I can do to make simple networks like this converge reliably? Or am I just approaching this with the wrong attitude, and you're supposed to just throw way more nodes than necessary at the problem and if half of them die it's not considered a big deal?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Your plot shows what appears to be several different experiments, some of which work and some don't. What accounts for the differences? $\endgroup$
    – Sycorax ♦
    Aug 22, 2018 at 17:04
  • $\begingroup$ @Sycorax Those are just repeated trials of the same thing. Sometimes it finds a hexagonal pyramid solution, but usually it doesn't. $\endgroup$
    – endolith
    Aug 22, 2018 at 18:57
  • $\begingroup$ Did you try with two hidden layers and a tanh as activation function? $\endgroup$
    – ksha
    Dec 2, 2018 at 8:45
  • $\begingroup$ @KetanNo, why? I posted a simpler variation here: stats.stackexchange.com/q/379884/11633 $\endgroup$
    – endolith
    Dec 2, 2018 at 15:12
  • $\begingroup$ you might want to look at this one. stats.stackexchange.com/q/375655/27556. But can you explain why you think you only need 3 relus? the 'natural' decomposition would be one hidden layer to do the squaring approximation with relus and another layer to do the square root - basically relus are doing piecewise linear approximations. $\endgroup$
    – seanv507
    Dec 2, 2018 at 16:01

1 Answer 1

Reset to default
7
$\begingroup$

The output seems to strongly suggest that one or more of your neurons goes dead (or perhaps the hyperplane of weights for two of your neurons have merged). You can see that with 3 Relu's, you get 3 shadowy splits in the center when you converge to the more reasonable solution. You can easily verify if this is true by checking the output values of each neuron to see if it stays dead for a large majority of your samples. Alternatively, you could plot all 2x3=6 neuron weights, grouped by their respective neuron, to see if two neurons collapse to the same pair of weights.

I suspect that one possible cause of this is when $x+iy$ is skewed toward one coordinate, e.g. $x\gg y$, in which case you're trying to reproduce the identity, as then $abs(x+iy)\approx x$. There's really not much you can do here to remedy this. One option is to add more neurons as you've tried. The second option is to try a continuous activation, like a sigmoid, or perhaps something unbounded like an exponential. You could also try dropout (with say, 10% probability). You could use the regular dropout implementation in keras, which is hopefully smart enough to ignore situations when all 3 of your neurons drop out.

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ +1, almost certainly this. When you use ReLU with such a small number of weights dead neurons almost always come into play. $\endgroup$
    – kbrose
    Aug 22, 2018 at 17:51
  • 1
    $\begingroup$ This is a plausible guess in the case of ReLU. But OP writes that they have tried a few different ReLU variants -- do dead neurons also occur in variants like ELU or Leaky Relu or PReLU? $\endgroup$
    – Sycorax ♦
    Aug 22, 2018 at 18:01
  • $\begingroup$ 3 shadowy splits in the center when you converge to the more reasonable solution. Yes, that's the crude approximation I meant; an inverted hexagonal pyramid. or perhaps something unbounded like an exponential I did try elu and selu, which didn't work much better. two neurons collapse to the same pair of weights Ah, I hadn't thought of that; I just assumed they were dead. $\endgroup$
    – endolith
    Aug 22, 2018 at 18:55
  • 1
    $\begingroup$ @endolith Dropout is explicitly designed around constructing independent neurons. ELUs never "die", but they do have vanishing gradient on the left. $\endgroup$
    – Sycorax ♦
    Aug 23, 2018 at 22:40
  • 2
    $\begingroup$ Your problem isn't overfitting, but dead neurons. When a neuron is dead, learning cannot proceed as the gradient is zero there, so you typically need to wait for extreme observations to activate that neuron. Even this isn't enough, as the gradient might still want the neuron to be even more dead due to other neurons being active. Dropout helps neurons be less dead on average by forcing each neuron to have comparable activation values (in their layer). Which means that when an extreme sample hits that neuron, it's unlikely to make it more dead, especially in your case as you only have 3. $\endgroup$
    – Alex R.
    Aug 24, 2018 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.