Why are nonlinear activations used in convolutional neural networks?

Question

Isn't convolution non-linear already? Do we need a nonlinearity in a different layer as as well? I understand the reasons why ReLU is good cited in this answer, but is there an intuitive reason we need nonlinear activation functions for CNNs?

Ways I've tried to answer this question:

1. How CNNs work? Stanford cs231.

2. In the paper Striving for Simplicity: The All Convolutional Net, Springenberg et al. suggest that pooling is bad and that we should strive for an all-convolutional neural net. Geoffrey Hinton also claims pooling is bad. Admittedly, I should read the paper more thoroughly. If I come up with a simple explanation after reading it, I'll post an answer.

3. Why do we use ReLU in neural networks and how do we use it?

4. What's the role of ReLU units in Convolutional neural networks? This question got closed for some reason, not sure why. It was a different question than the question they claimed was duplicate.

5. Why must a nonlinear activation function be used in a backpropagation neural network

Answer 1

Given the weight matrix, the convolution is a linear operator. If $x$ and $y$ are image vectors, and $W$ is the weight matrix, then it is easy to verify that

$f(x+y;W)$ = $f(x;W) + f(y;W)$ --> (intuition is that I can either convolve over 2 different images and then add them, or vice versa)

and

$f(ax;W) = af(x;W)$ --> (intuition is that if I amplify each pixel in the image by a factor, then the entire convolution will be amplified by that factor)