10
$\begingroup$

As the title suggests, I have several features which have values of either -1, 0 or 1. If I feed this data into a neural network where I use ReLu as the activation function for the hidden layers, would the negative and 0 values pose a problem to the NN?

I have heard about dead neurons where using ReLu which is a stepwise function, causes any inputs less than or equal to 0 the neuron to stop learning and become dead. So naturally if a NN with activation function ReLu is fed 0 or negative inputs, those neurons will become dead.

Now my data contains several features with 0 and negative values. What to do in such a case? Should I use LeakyReLu or some other variation of ReLu? Or should I transform my data such that only positive values remain?

EDIT 1: If the negative and 0 inputs do not cause dead neurons then what causes dead neurons? Also then why do we have activation functions like LeakyReLu, PReLu, ELU if ReLu alone can handle dead neurons?

Improve this question
$\endgroup$
2
  • 8
    $\begingroup$ It's in fact very common to normalize your inputs to zero mean, standard deviation 1.0. Zero mean obviously means that some outputs are negative, typically about half. $\endgroup$
    – MSalters
    Dec 22, 2021 at 20:13
  • 4
    $\begingroup$ I believe that EDIT 1 should be a separate question. $\endgroup$
    – Dave
    Dec 23, 2021 at 12:34

5 Answers 5

Reset to default
9
$\begingroup$
  • (this has been said already by other answers) ReLu activation function has a gradient equal to 0 (hence it stops learning) when the linear combination of the inputs is less than 0, not when the input themselves are 0
  • (this also has been said already by other answers) the inputs of NNs are often normalised around 0, so it's totally normal that some values are 0, also the weights of each neuron are usually (randomly) initialised around 0, meaning that when it comes the time to compute the linear combination, some random input values will switch sign, this is expected
  • ReLu function is actually designed to result in a null gradient for values below 0, don't stress out about this, the problem of dead neurons comes up when all inputs for that neuron result in a null gradient. It's not a trivial problem to discuss so I will slide upon it, but it has nothing to do with simply having some negative values in the inputs. As HitLuca has pointed in his comment, having the neuron parameters go to zero during the learning process will cause the neuron to die.
  • Of course other activation functions that never result in a null gradient (like leaky ReLu) will avoid dead neurons entirely.
Improve this answer
$\endgroup$
0
17
$\begingroup$

It's not your inputs which are passed to the activation function (ReLu in your case), your inputs are first transformed using a linear (affine) transformation, and these transformed values are then passed to the activation function. So negative values are not an issue, it is entirely possible that negative values will be transformed to positive values.

Improve this answer
$\endgroup$
0
8
$\begingroup$

LeakyRelu is a good option for remedying dead neurons, so it's reasonable to use. But, having negative inputs doesn't cause this because your weight initialization can also have negative weights and turn the positive inputs into negative when multiplied. This is hardly a problem. It's usually preferred that your inputs have zero-mean, so very natural to have negative inputs as well.

Improve this answer
$\endgroup$
0
3
$\begingroup$

Dead neurons refer to the fact that the network comes to a state where the back-propagation step (which is used to update the internal network's weights and biases) isn't effective enough, as the changes in values are zero or close to zero. One way to get a dead neuron situation is to initialize your network with weight and bias values of zero, with ReLU activations: this will produce an input value to the activation function of zero, which has a derivative of zero in that point. When updating the weights, the gradient will be zero so no changes are made.

The same issue can arise when a neuron received negative values to its ReLU activation function: since for x<=0 f(x)=0, the output will always be zero, with again zero gradient.

To remedy this, we have activation functions like LeakyReLU, which has nonzero activation for negative values, and thus nonzero gradient. A zero-initialized network will still have the same issue, as LeakyReLUs still have outputs of zero for any input, but manage to solve the problems with negative values.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Nice explanation! But the other answers seem to disagree with you. They say negative inputs do not produce dead neurons and neither do zero inputs! $\endgroup$
    – spectre
    Dec 23, 2021 at 10:39
  • 1
    $\begingroup$ @spectre that is true, but I mentioned the fact that what may produce dead neurons is having zero weights and biases, meaning you would end up doing ReLU(input*0+0) -> always 0 $\endgroup$
    – HitLuca
    Dec 23, 2021 at 11:33
  • $\begingroup$ Ok got it now! The weights and biases both being null will cause dead neurons! $\endgroup$
    – spectre
    Dec 23, 2021 at 12:47
  • $\begingroup$ Technically being 0 (not null), there are some other cases in which it can happen, but it's a very straightforward example to understand why this stuff happens :) $\endgroup$
    – HitLuca
    Dec 23, 2021 at 13:05
2
$\begingroup$

ReLu neurons can indeed effectively ignore some input values. This is a good thing. The neurons are coupled to the inputs via trained weights, so some inputs are ignored by some neurons and other inputs are ignored by other neurons. It allows weights in the subsequent layers to focus on interesting subsets of your input space

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.