Timeline for Why is ReLU setting negative values to zero particularly?
Current License: CC BY-SA 3.0
5 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Jan 12, 2023 at 16:04 | comment | added | Frans Rodenburg | You only 'kill negative values' if you have a bias term of zero and positive weights. More accurately, you limit the range of the input input on either side, at some value that can be learned through training. | |
Jan 5, 2023 at 17:40 | comment | added | Vishal | This doesn't actually answer the question; why are we choosing activations functions that essentially kill negative values (this include ReLU/GeLU et al)? Saying that we need non-linear activations functions isn't an answer, as there exist an infinite number of activation functions that are differentiable, etc. | |
Apr 1, 2017 at 12:54 | comment | added | Yuval Filmus | Without non-linearity, your network will compute just some linear function. No need to make it deep or anything. | |
Apr 1, 2017 at 7:23 | comment | added | jsdbt | Can I keep it as $x$ only? Sparsity can anyway be induced by dropout. (PS Ignoring the non-linearity that $max$ or $min$ form would introduce in the system) | |
Apr 1, 2017 at 6:48 | history | answered | AaronDefazio | CC BY-SA 3.0 |