When I first started out learning neural-networks, I tried to get intuition for why they work with logit activation functions. I pictured each "neuron" as doing a logistic regression on the layer below in order to model the binomial distribution of "Is the feature that this neuron represents present, given the layer below? 1 for yes, 0 for no." Through gradient descent, each neuron converges on a feature that is most useful for the network to recognize.

When moving on to other activation functions, particularly the ReLU, my intuition falls apart, because now you're not doing logistic regression on the layer below. You're no longer using the output below to model a binomial distribution. So what are you really doing? How does the ReLU activation still "recognize" features that are lower in the hierarchy?

  • $\begingroup$ if you look at Relu activation function graph, you'll see that it a threshold above zero which allow data to pass since it over this threshold. in comparison wit the sigmoid squashed function. it is better in derivative part since the saturation part is only on the area under zero. it is still representing values of the layer below by the thresholding in a larger range instead of zero or one. $\endgroup$ – Feras Aug 15 '17 at 12:58
  • $\begingroup$ There is a paper by Hinton which shows that a bunch of ReLU are identical to using a bunch of sigmoids so your intuition works. Anyhow, I would advise you to modify your intuition. This chapter has javascript interactive activation functions which can help you with that. $\endgroup$ – Ricardo Cruz Aug 17 '17 at 9:45

Imagine running linear regression when you expect the results to always be positive. Therefore, even if the prediction is negative, you set it to 0 to get a valid output, so effectively, $y = \text{relu}(w^Tx)$. Now if you simply "stack" these linear regression units in the same way you "stack" logistic regression to get a neural network, you end up with a neural network using relu units.

Another way to see why relu works is to drop the idea of sigmoid units doing logistic regression -- because they're not really doing logistic regression in any traditional sense. Instead, the sum total of the neural network is acting as a powerful function approximator. It has been shown that a neural network of sufficient size can approximate almost any function arbitrarily well. We want to train our neural network to approximate the function which maps the inputs to the correct outputs.

When you think about a neural network as a function approximator, it makes sense that relu works just as well as sigmoid -- they both play the role of introducing non-linearities into the network (which is required for the universal approximation theorem to hold).

To sum it up, you can replace logistic regression with a modified form of linear regression to satisfy your intuition. However, viewing the network as a function approximator may be a better way to see how neural networks work.

  • $\begingroup$ I'm up to speed on the goal of the whole network, so I'm totally on board with the function approximator concept. I guess what I'm really asking is why does stacking a bunch of non linearities together work to achieve that goal? And why doesn't it seem to matter what non linearity you use? I mean it does algorithmically, but it works acceptable with both logit and relu, and those aren't very similar non linearities $\endgroup$ – Cordello Aug 17 '17 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.