I want to understand the logic behind keeping ReLU as $max(0,x)$ and not $min(0,x)$?

Why do we prefer positive inputs over the negative ones?


The weights learned in a neural network can be both positive and negative. So in effect, either form would work. Negating the input and output weights with the $\min$ form gives the same function as with the $\max$ form. The max form is used purely by convention.

  • $\begingroup$ Can I keep it as $x$ only? Sparsity can anyway be induced by dropout. (PS Ignoring the non-linearity that $max$ or $min$ form would introduce in the system) $\endgroup$ – jsdbt Apr 1 '17 at 7:23
  • 2
    $\begingroup$ Without non-linearity, your network will compute just some linear function. No need to make it deep or anything. $\endgroup$ – Yuval Filmus Apr 1 '17 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.