Isn't convolution non-linear already? Do we need a nonlinearity in a different layer as as well? I understand the reasons why ReLU is good cited in this answer, but is there an intuitive reason we need nonlinear activation functions for CNNs?
Ways I've tried to answer this question:
How CNNs work? Stanford cs231.
In the paper Striving for Simplicity: The All Convolutional Net, Springenberg et al. suggest that pooling is bad and that we should strive for an all-convolutional neural net. Geoffrey Hinton also claims pooling is bad. Admittedly, I should read the paper more thoroughly. If I come up with a simple explanation after reading it, I'll post an answer.
What's the role of ReLU units in Convolutional neural networks? This question got closed for some reason, not sure why. It was a different question than the question they claimed was duplicate.