0
$\begingroup$

I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My question is when all of the numbers in neural network nodes are positive , relu function becomes a linear function itself. In such a situation how does it create nonlinearity and help the neural network to actually learn a nonlinear combination of input features ?

$\endgroup$

1 Answer 1

0
$\begingroup$

The behavior will depend on the inputs, $x$. If some of the features in $x$ are negative, then this network is still nonlinear because a negative number multiplied by a positive number is negative, so the ReLU will be nonlinear.

$\endgroup$
2
  • $\begingroup$ However;in a situation where all features are positive, it became meaningless to use relu right ? Or can there be some features will create negative values after backpropagation ? $\endgroup$
    – levitatmas
    Apr 15, 2022 at 20:30
  • $\begingroup$ Are you asking about a network with fixed values of its weights and biases or a network that is being trained? If you're training the network, then the weights can change, so weights that start off as positive may not be positive at the end of training. $\endgroup$
    – Sycorax
    Apr 15, 2022 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.