When searching for information on choosing the number of hidden layers in a neural network, I have come across the following table mutiple times, including in this answer:

| Number of Hidden Layers | Result |

0 - Only capable of representing linear separable functions or decisions.

1 - Can approximate any function that contains a continuous mapping from one finite space to another.

2 - Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.

I am familiar with the universal approximation theorem for 1 hidden layer, but not with the purported result about the additional power of 2 hidden layers. Is it true? If so, where can I find a detailed explanation and proof?

Edit: Apparently the table comes from Jeff Heaton.

  • $\begingroup$ Perhaps not that important, but I think the key intuition would be the composition of non-linear activation functions. With no hidden layers, you have one activation function on the output. With one hidden layer, you now have one "internal" non-linear activation function and one after your output node. (Assuming a regression setting here.) With two hidden layers you now have an internal "composition" (may be misusing the term here) of two non-linear activation functions. $\endgroup$ – Wayne Nov 19 '17 at 17:43

I found the answer to my question in the paper Feedback stabilization using two-hidden-layer nets by E.D. Sontag. From the introduction:

It is by now well-known that functions computable by nets with a single hidden layer can approximate continuous functions, uniformly on compacts, under only weak assumptions on $\theta$. Consider now the following inversion problem: Given a continuous function $f : \mathbb{R}^m \rightarrow \mathbb{R}^p$, a compact subset $C \subseteq \mathbb{R}^p$ included in the image of $f$, and an $\varepsilon > 0$, find a function $\phi : \mathbb{R}^p \rightarrow \mathbb{R}^m$ so that $\|f(\phi(x)) - x \| < \varepsilon$ for all $x \in C$. It is trivial to see that in general discontinuous functions $\phi$ are needed. We show later that nets with just one hidden layer are not enough to guarantee the solution of all such problems, but nets with two hidden layers are.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.