1
$\begingroup$

The universal approximation theorem states that a feedforward neural network (NN) with a single hidden layer can approximate any function over some compact set, provided that it has enough neurons on that layer.

This suggests that the number of neurons is more important than the number of layers.

But in practice deep learning is obviously very successful at various prediction tasks. Why is that? Shouldn't all deep NNs be equivalent to single layered NNs with enough neurons? Why do we need depth?

$\endgroup$
1
  • $\begingroup$ You may want to take a look at this article. $\endgroup$
    – agcala
    Apr 14, 2019 at 11:39

2 Answers 2

2
$\begingroup$

Neural networks (kind of) need multiple layers in order to learn more detailed and more abstractions relationships within the data and how the features interact with each other on a non-linear level.

Even though it is theoretically possible to represent any possible function with a single hidden layer neural network, determine the number of nodes needed in that hidden layer is difficult. Therefore, adding more layers (apart from increasing computational complexity to the training and testing phases), allows for more easy representation of the interactions within the input data, as well as allows for more abstract features to be learned and used as input into the next hidden layer.

from David Torpy

Basically, it makes your network more eager to recognize certain aspects of input data. For example, if you have the details of a house (big house, size, etc.) as input and want to predict the price. The first layer may predict:

  • Big area, higher price
  • Small amount of bedrooms, lower price

The second layer might conclude:

  • Big area + small amount of bedrooms = large bedrooms = +- effect

Yes, one layer can also 'detect' the stats, however it will require more neurons as it cannot rely on other neurons to do 'parts' of the total calculation required to detect that stat.

$\endgroup$
1
$\begingroup$

If you have a single layer, the number of neurons you need, to match a multi-layer network, is combinatorially large.

With multiple layers, each layer can create more and more abstract features/concepts. Like, in our brain, the output from our eyes enters the brain at the back, and passes through very low-level feature detectors. Lines and such. As the signal moves forward through the layers, the features become more abstract, from simple line detectors to various types of moving object and so on.

So it is with nets. Depth gives the possibility of abstracting relatively abstract concepts, in the upper layers, which massively improves the ability of the network to classify and so on. As long as there is sufficient data.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.