# Why is ReLU setting negative values to zero particularly?

I want to understand the logic behind keeping ReLU as $max(0,x)$ and not $min(0,x)$?

Why do we prefer positive inputs over the negative ones?

The weights learned in a neural network can be both positive and negative. So in effect, either form would work. Negating the input and output weights with the $\min$ form gives the same function as with the $\max$ form. The max form is used purely by convention.

• Can I keep it as $x$ only? Sparsity can anyway be induced by dropout. (PS Ignoring the non-linearity that $max$ or $min$ form would introduce in the system) – jsdbt Apr 1 '17 at 7:23
• Without non-linearity, your network will compute just some linear function. No need to make it deep or anything. – Yuval Filmus Apr 1 '17 at 12:54