# Why would ReLU work as an activation function at all?

When I first started out learning neural-networks, I tried to get intuition for why they work with logit activation functions. I pictured each "neuron" as doing a logistic regression on the layer below in order to model the binomial distribution of "Is the feature that this neuron represents present, given the layer below? 1 for yes, 0 for no." Through gradient descent, each neuron converges on a feature that is most useful for the network to recognize.

When moving on to other activation functions, particularly the ReLU, my intuition falls apart, because now you're not doing logistic regression on the layer below. You're no longer using the output below to model a binomial distribution. So what are you really doing? How does the ReLU activation still "recognize" features that are lower in the hierarchy?

• if you look at Relu activation function graph, you'll see that it a threshold above zero which allow data to pass since it over this threshold. in comparison wit the sigmoid squashed function. it is better in derivative part since the saturation part is only on the area under zero. it is still representing values of the layer below by the thresholding in a larger range instead of zero or one. – Feras Aug 15 '17 at 12:58
• There is a paper by Hinton which shows that a bunch of ReLU are identical to using a bunch of sigmoids so your intuition works. Anyhow, I would advise you to modify your intuition. This chapter has javascript interactive activation functions which can help you with that. – Ricardo Cruz Aug 17 '17 at 9:45

Imagine running linear regression when you expect the results to always be positive. Therefore, even if the prediction is negative, you set it to 0 to get a valid output, so effectively, $y = \text{relu}(w^Tx)$. Now if you simply "stack" these linear regression units in the same way you "stack" logistic regression to get a neural network, you end up with a neural network using relu units.