# Why would ReLU work as an activation function at all?

When I first started out learning neural-networks, I tried to get intuition for why they work with logit activation functions. I pictured each "neuron" as doing a logistic regression on the layer below in order to model the binomial distribution of "Is the feature that this neuron represents present, given the layer below? 1 for yes, 0 for no." Through gradient descent, each neuron converges on a feature that is most useful for the network to recognize.

When moving on to other activation functions, particularly the ReLU, my intuition falls apart, because now you're not doing logistic regression on the layer below. You're no longer using the output below to model a binomial distribution. So what are you really doing? How does the ReLU activation still "recognize" features that are lower in the hierarchy?

• if you look at Relu activation function graph, you'll see that it a threshold above zero which allow data to pass since it over this threshold. in comparison wit the sigmoid squashed function. it is better in derivative part since the saturation part is only on the area under zero. it is still representing values of the layer below by the thresholding in a larger range instead of zero or one. – Feras Aug 15 '17 at 12:58
• There is a paper by Hinton which shows that a bunch of ReLU are identical to using a bunch of sigmoids so your intuition works. Anyhow, I would advise you to modify your intuition. This chapter has javascript interactive activation functions which can help you with that. – Ricardo Cruz Aug 17 '17 at 9:45
• For a demonstration, see stats.stackexchange.com/questions/299915/… – Sycorax Feb 3 '20 at 16:37

Imagine running linear regression when you expect the results to always be positive. Therefore, even if the prediction is negative, you set it to 0 to get a valid output, so effectively, $y = \text{relu}(w^Tx)$. Now if you simply "stack" these linear regression units in the same way you "stack" logistic regression to get a neural network, you end up with a neural network using relu units.

Another way to see why relu works is to drop the idea of sigmoid units doing logistic regression -- because they're not really doing logistic regression in any traditional sense. Instead, the sum total of the neural network is acting as a powerful function approximator. It has been shown that a neural network of sufficient size can approximate almost any function arbitrarily well. We want to train our neural network to approximate the function which maps the inputs to the correct outputs.

When you think about a neural network as a function approximator, it makes sense that relu works just as well as sigmoid -- they both play the role of introducing non-linearities into the network (which is required for the universal approximation theorem to hold).

To sum it up, you can replace logistic regression with a modified form of linear regression to satisfy your intuition. However, viewing the network as a function approximator may be a better way to see how neural networks work.

• I'm up to speed on the goal of the whole network, so I'm totally on board with the function approximator concept. I guess what I'm really asking is why does stacking a bunch of non linearities together work to achieve that goal? And why doesn't it seem to matter what non linearity you use? I mean it does algorithmically, but it works acceptable with both logit and relu, and those aren't very similar non linearities – Cordello Aug 17 '17 at 3:12
• Except the standard proof of the universal approximation theorem relies on the boundedness of the activation functions. There are extensions, but the fact that ReLU works is not obvious to me. – Abhimanyu Pallavi Sudhir Aug 2 '19 at 16:50

I know this is an old post but it still pops up on Google when you search for intuitions so I would like to add the results of my own questioning.

The idea that sigmoid was good for binomial distribution was good for intuition when discovering it, but in fact, with deep learning, we combine so many neurons on so many layers that we are not really looking for binomial distribution, not for each separated neuron! That's why we can move on to shimao's idea of function approximator.

Then for the ReLu idea: it approaches very closely how our own brain works! Our neurons propagate an action potential (a peak of excitation) only when the excitation (information from the synapse) has reached a threshold level. If the neuron has received some information from the synapse but not enough, it doesn't do anything. That vision helped me accept that ReLu might be a good idea!