Isn't convolution non-linear already? Do we need a nonlinearity in a different layer as as well? I understand the reasons why ReLU is good cited in this answer, but is there an intuitive reason we need nonlinear activation functions for CNNs?

Ways I've tried to answer this question:

  1. How CNNs work? Stanford cs231.

  2. In the paper Striving for Simplicity: The All Convolutional Net, Springenberg et al. suggest that pooling is bad and that we should strive for an all-convolutional neural net. Geoffrey Hinton also claims pooling is bad. Admittedly, I should read the paper more thoroughly. If I come up with a simple explanation after reading it, I'll post an answer.

  3. Why do we use ReLU in neural networks and how do we use it?

  4. What's the role of ReLU units in Convolutional neural networks? This question got closed for some reason, not sure why. It was a different question than the question they claimed was duplicate.

  5. Why must a nonlinear activation function be used in a backpropagation neural network

  • $\begingroup$ "Isn't convolution non-linear already?" No, a convolution is a linear operation. $\endgroup$ Commented Apr 1 at 3:12

1 Answer 1


Given the weight matrix, the convolution is a linear operator. If $x$ and $y$ are image vectors, and $W$ is the weight matrix, then it is easy to verify that

$f(x+y;W)$ = $f(x;W) + f(y;W)$ --> (intuition is that I can either convolve over 2 different images and then add them, or vice versa)


$f(ax;W) = af(x;W)$ --> (intuition is that if I amplify each pixel in the image by a factor, then the entire convolution will be amplified by that factor)

  • 2
    $\begingroup$ However, most CNNs also apply maxpooling, which is non-linear. Why would activation functions be necessary in this case? I can imagine that immediately stacked convlayers, for example in a residual unit, can benefit from activations to add non-linearity and not just the visible range. But consider a relu and maxpooling, why add the relu that can only kill neurons, when you already can get an elbow from the maxpool? $\endgroup$
    – Herbert
    Commented Nov 18, 2021 at 13:32
  • $\begingroup$ @Herbert Maybe max pooling alone isn't sufficient to make a convolutional neural network a universal function approximator? $\endgroup$ Commented Apr 1 at 3:21
  • 1
    $\begingroup$ @HelloGoodbye Is your first comment a question or a statement? Regarding your second comment, that's what I meant with "immediately stacked convlayers". $\endgroup$
    – Herbert
    Commented Apr 3 at 8:39
  • 1
    $\begingroup$ @Herbert It is a question, since I'm not sure about the answer. But if it isn't enough, then maybe that's the answer to your question. If you don't have a network that is a universal function approximator, that means it can't model any function you want to be able to model, which typically makes it significantly less powerful than a network that is a universal function approximator. $\endgroup$ Commented Apr 28 at 19:27
  • 1
    $\begingroup$ @HelloGoodbye max-pool runs over something that's highly correlated, e.g. a sliding window dot-product over neighbors. So you might be very right that this doesn't add up to a universal function approximator. $\endgroup$
    – Herbert
    Commented Apr 29 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.