2 hidden layers are more powerful than 1

Question

When searching for information on choosing the number of hidden layers in a neural network, I have come across the following table mutiple times, including in this answer:

| Number of Hidden Layers | Result |

0 - Only capable of representing linear separable functions or decisions.

1 - Can approximate any function that contains a continuous mapping from one finite space to another.

2 - Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.

I am familiar with the universal approximation theorem for 1 hidden layer, but not with the purported result about the additional power of 2 hidden layers. Is it true? If so, where can I find a detailed explanation and proof?

Edit: Apparently the table comes from Jeff Heaton.

Answer 1

I found the answer to my question in the paper Feedback stabilization using two-hidden-layer nets by E.D. Sontag. From the introduction:

It is by now well-known that functions computable by nets with a single hidden layer can approximate continuous functions, uniformly on compacts, under only weak assumptions on $\theta$. Consider now the following inversion problem: Given a continuous function $f : \mathbb{R}^m \rightarrow \mathbb{R}^p$, a compact subset $C \subseteq \mathbb{R}^p$ included in the image of $f$, and an $\varepsilon > 0$, find a function $\phi : \mathbb{R}^p \rightarrow \mathbb{R}^m$ so that $\|f(\phi(x)) - x \| < \varepsilon$ for all $x \in C$. It is trivial to see that in general discontinuous functions $\phi$ are needed. We show later that nets with just one hidden layer are not enough to guarantee the solution of all such problems, but nets with two hidden layers are.

Answer 2

Another paper about a qualitative difference between 1- and 2-hidden layer networks:

Neural networks for localized approximation (1994)

We prove that feedforward artificial neural networks with a single hidden layer and an ideal sigmoidal response function cannot provide localized approximation in a Euclidean space of dimension higher than one. We also show that networks with two hidden layers can be designed to provide localized approximation.

The objective of this paper is to investigate the possibility of constructing networks suitable for localized approximation, i.e., a network with the property that if the target function is modified only on a small subset of the Euclidean space, then only a few neurons, rather than the entire network, need to be retrained... We prove that if the dimension of the input space is greater than one, then such a network with one hidden layer and a Heaviside activation function cannot be constructed. In contrast, we also show that a network with two or more hidden layers can always be constructed to accomplish the task.

AMS link