I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My question is when all of the numbers in neural network nodes are positive , relu function becomes a linear function itself. In such a situation how does it create nonlinearity and help the neural network to actually learn a nonlinear combination of input features ?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The behavior will depend on the inputs, $x$. If some of the features in $x$ are negative, then this network is still nonlinear because a negative number multiplied by a positive number is negative, so the ReLU will be nonlinear.
answered Apr 15, 2022 at 20:27
-
$\begingroup$ However;in a situation where all features are positive, it became meaningless to use relu right ? Or can there be some features will create negative values after backpropagation ? $\endgroup$ Apr 15, 2022 at 20:30
-
$\begingroup$ Are you asking about a network with fixed values of its weights and biases or a network that is being trained? If you're training the network, then the weights can change, so weights that start off as positive may not be positive at the end of training. $\endgroup$– Sycorax ♦Apr 15, 2022 at 20:33