When I first started out learning neural-networks, I tried to get intuition for why they work with logit activation functions. I pictured each "neuron" as doing a logistic regression on the layer below in order to model the binomial distribution of "Is the feature that this neuron represents present, given the layer below? 1 for yes, 0 for no." Through gradient descent, each neuron converges on a feature that is most useful for the network to recognize.
When moving on to other activation functions, particularly the ReLU, my intuition falls apart, because now you're not doing logistic regression on the layer below. You're no longer using the output below to model a binomial distribution. So what are you really doing? How does the ReLU activation still "recognize" features that are lower in the hierarchy?