Advantages of convolutional neural networks over "simple" feed-forward networks?

Question

How are convolutional networks better than simple neural networks (Feed-forward networks)? Is the convolution operation the only difference between convolutional neural networks and deep networks?

Answer 1

Any time that you can legitimately make stronger assumptions, you can obtain stronger results. Convolutional networks make the assumption of locality, and hence are more powerful. This depends on data that in fact exhibits locality (autocorrelation) like images or time series.

Intuitively, if you are looking at an image, pixels in a region of the image are more likely to be related than pixels far away. So you can save a lot of neuron wiring if you don't (directly) wire distant pixels to the same neuron. With less wiring, you have more data per coefficient, which speeds things up and makes for better results.

Answer 2

The sentence said by @Wayne summarizes it pretty well - "Any time that you can legitimately make stronger assumptions, you can obtain stronger results". Though, a bit more detail can be added to why CNNs are superior models for image data compared to the multi-layer perceptron.

Fully connected layers are the fundamental components of a multiplayer perceptron, the simplest form of an artificial neural network. Often times, the final component in a convolutional neural network (CNN) is a fully connected layer because it performs the regression or classification task.

The components at the start of a neural network typically serve the purpose of reducing the inputs to a more digestible form for classification, regression, etc. In the context of a CNN, one of the most important components is the convolution operation and its job is to reduce the image to a more suitable form.

For the convolution operation, the input image is convolved with a filter to produce a new image, called a feature map, which reveals the parts of the input that are most similar to the filter. For example, say we want to automate the detection of hemocyanin particles in microscopy images via a CNN model. One thing a CNN might learn to do is to convolve the input image with a filter that looks like a hemocyanin particle. The pixels in the feature map will be brighter in the regions of the image that contain hemocyanin particles.

It's pretty straight forward to see why the reduced image is more useful for detection than the original. This is the advantage of a CNN.

Answer 3

Deep nets are a general class whose solely mandatory characteristic is an unusual number of neural layers, whereas convolutional networks are a specific technique that can be included in a deep net, alongside other techniques such as LSTMs, perceptrons, Kohonen nets, etc. Keep in mind that "deep learning" is something of a buzz word with a fuzzy definition; in practice, these other neural techniques aren't included anywhere near as often in the equally fuzzy definition of deep nets. I believe that most "deep" networks consist mainly of convolutional layers, based on the academic papers I've read so far. The term doesn't necessarily imply or demand the inclusion of convolutionals in every deep network - you could, for example, have a deep net consisting of two dozen perceptron layers, although that would hardly be practical. There are practical advantages, however, to piling convolutionals on top of each other in stacks, particularly the ability to leverage convolution filters to create hierarchical representations. There may be other subtle advantages to convolutionals, but that is by far the dominant one. I'll leave it to more experienced users to provide the mathematical and logical details as to why convolutionals can be stacked to create hierarhical representations, but suffice it to say that this is the "short form" answer to your questions: 1) they're better mainly because of the convolutional filtering operation and 2) is something of an apples and oranges comparison, since one is a general class and the other is a specific type often included within it.