5
$\begingroup$

I've been reading Chris Bishop's book, Neural Networks for Pattern Recognition recently. It is quite dated relative to the state of the art, but it is the best book I've found on neural nets that actual relates things back to statistics, optimization, etc.

Reading it today, I crossed a passage that I've heard before, specifically: a three layer network (2 hidden layers, or 3 layers of weights) with sigmoidal activation functions can approximate, to arbitrary accuracy, any smooth mapping. He goes on further to say that given an adequate number of hidden units, dimensionality of input to output mapping is not limiting. I believe also he describes a 3 layer network as a universal function approximator.

Also, I recently read a paper (https://arxiv.org/abs/1806.06850) showing empirically that polynomial regression was able to perform as well or better than neural networks of depth similar to the degree of the polynomial, but, they only explored to the power of 3.

With all the interest in deep neural networks, I'm wondering why tall neural networks haven't also been of interest. Obviously the name isn't as sexy, but it seems in general I could have a more shallow network that is simply taller and get to the same result. The only thing that comes to mind for me is the gradients for each layer are much higher dimension, and maybe convergence would be a problem.

Or I'm missing something more deep. I appreciate any thoughts on the topic.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Because complexity grows exponentially with depth, as opposed to polynomially with width. That paper's results are restricted to 3 layer networks, which is laughably low for any deep learning application. Contrast that with deep networks on images, which have hundreds of layers. $\endgroup$
    – Alex R.
    Jul 23, 2018 at 23:12
  • 3
    $\begingroup$ So the "height" or "width" of a 3 layer network could approach the same capability of a deep network, but the number of nodes required by the "wide" network would become computationally intractable to achieve the same complexity of the comparative deep network. Does that capture it? $\endgroup$
    – KirkD_CO
    Jul 23, 2018 at 23:49

1 Answer 1

Reset to default
3
$\begingroup$

Whilst a 3 layer network can be assigned weights that can approximate any function, this function might not be easily learnable via backpropagation.

Whereas deep networks can learn effectively from back-propagation.

Just to show that it is in fact possible to pack a lot of the learning from a deep network into a shallow network, you can look at 'Do deep nets really need to be deep", https://arxiv.org/abs/1312.6184, by Ba and Caruna. In this paper, they show that a shallow network that cannot learn a good fit to the desired function from supervised training on the original task, can in fact learn this function by being trained from a deep 'teacher' network.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.