1
$\begingroup$

Sorry I'm a beginer. I understand the nature of non-linear vs linear activation functions, I know RELU basically filter the negatives inputs and only respond to the positive, but When does it happen that a layer (either first or hidden) outputs negatives values in order to justify the use of RELU? As far as I know features are never negative or converted to negative in any other type of layer. Is it that we can use the RELU with a different "inflection" point than zero? so we can make the neuron start describing a lineal response just after this "new zero". Please correct me, thanks.

$\endgroup$
1
$\begingroup$

Weights and biases of neutral networks are not constrained, so they can be either positive, or negative. If you have features that are positive, if you multiply them by negative numbers, then the outputs that will be passed to activation function will be negative. As a comment, this is something that you probably already know, but the point of using ReLU is not to make things positive, but to introduce non-linearity in the network. You could use $\min(0, z)$ as activation function as well and to convert the predictions to positive values, weights on final layer would need to be negative, or biases big enough.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The output of a neuron cannot be negative because of the ReLU activation function. ReLU is used address the problem of vanishing gradient (Refer here) and introduce the non-linearity in the model, how?. Because we can approximate very complex curves using the combination of different ReLU units.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.