Skip to main content
5 events
when toggle format what by license comment
Jan 12, 2023 at 16:04 comment added Frans Rodenburg You only 'kill negative values' if you have a bias term of zero and positive weights. More accurately, you limit the range of the input input on either side, at some value that can be learned through training.
Jan 5, 2023 at 17:40 comment added Vishal This doesn't actually answer the question; why are we choosing activations functions that essentially kill negative values (this include ReLU/GeLU et al)? Saying that we need non-linear activations functions isn't an answer, as there exist an infinite number of activation functions that are differentiable, etc.
Apr 1, 2017 at 12:54 comment added Yuval Filmus Without non-linearity, your network will compute just some linear function. No need to make it deep or anything.
Apr 1, 2017 at 7:23 comment added jsdbt Can I keep it as $x$ only? Sparsity can anyway be induced by dropout. (PS Ignoring the non-linearity that $max$ or $min$ form would introduce in the system)
Apr 1, 2017 at 6:48 history answered AaronDefazio CC BY-SA 3.0