If convolution can be expressed with matrix multiplication (example) Can we say convolution neural network (CNN) is a special case of multilayer perceptron (MLP)?

If yes, why people do not use a big enough MLP for everything, that let the computer to learn to use the convolution by self?


1 Answer 1


A convolution can be expressed as matrix multiplication but the matrix is multiplied with a patch around every position in the image separately. So you go to (1/1) and extract a patch and multiply it with an MLP. Then you do the same thing at position (1/2) and so forth. So obviously there are less degrees of freedom than applying an MLP directly. Most people regard an MLP as a special case of a convolution where the spatial dimensions are 1x1.

Edit Start

Regarding MLP as special case of CNN, some comments do not share this opinion. Yann LeCun, who can be counted as one of the inventors of CNNs, made a similar comment before on FB: https://www.facebook.com/yann.lecun/posts/10152820758292143

He said that in CNNs there is no such thing as a "fully connected" layer, there is only a layer with 1x1 spatial extent and a kernel with 1x1 spatial extent. If one can "convert" FC layers, which are the single layers of MLPs into convolutional layers, then one can obviously also convert an entire MLP into a CNN by interpreting the input as a vector with only channel dimensions.

An example: If I have an image of size $H\times W\times C$ ($C$ channels) and I apply a single layer of an MLP to it, then I will transform the input into a vector $x$ of size $V=HWC$. I will then apply a matrix $W\in \mathbb{R}^{U\times V}$ to it, thereby creating $U$ hidden activations. I could interpret the input vector $x$ as an image with only one pixel but $V$ "channels": $x\in\mathbb{R}^{1\times 1\times V}$ and the weight matrix as a Kernel with only one pixel area but $U$ filters taking in $V$ channels each: $W\in\mathbb{R}^{U\times 1\times 1\times V}$. I can then call some Conv2D function that carries out the operation and computes exactly the same as the MLP.

Edit End

If yes, why people do not use a big enough MLP for everything, that let the computer to learn to use the convolution by self?

That is a nice idea (and probably worth doing research on) but it's simply not practical:

  1. The MLP has too many degrees of freedom, it's likely to overfit.
  2. In addition to learning the weights, you would have to learn their dependency structure.

As most Deep Learning research is closely related to NLP/speech processing/computer vision, people are eager to solve their problems and maybe less eager to investigate how a function space more general than a CNN could constrain itself to that particular function space. Though imho it's certainly interesting to think about that.

  • $\begingroup$ I am not sure if that is the case ("people regard an MLP as a special case of a convolution"). Another equally valid way of looking at it is that a CNN is a special case of a MLP where only local connections have a weight different from zero, and that the weights of local connections are shared. Definitely that is how I was introduced to the concept of CNNs after learning about fully-connected networks. $\endgroup$ Aug 26, 2018 at 11:50
  • $\begingroup$ @NeilSlater I've edited the post and tried to clarify my view on that matter, I hope it's now easier to understand. If you could take the time to share your point of view, that would be very much appreciated, as I think the question is quite interesting. Thank you! $\endgroup$
    – jmaxx
    Aug 26, 2018 at 15:56
  • $\begingroup$ Yann LeCun's comment also leads to this Q&A on Stack Exchange: datascience.stackexchange.com/questions/12830/… $\endgroup$ Aug 26, 2018 at 16:05
  • $\begingroup$ @NeilSlater Thanks for the link, I hadn't seen that there is an answer on another Stack Exchange site. As far as I can tell, that answer gives a similar perspective as mine in that it shows how any MLP can be computed as a $1\times 1$ convolution, i.e. those two are equivalent. $\endgroup$
    – jmaxx
    Aug 26, 2018 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.