# 2 hidden layers are more powerful than 1

When searching for information on choosing the number of hidden layers in a neural network, I have come across the following table mutiple times, including in this answer:

| Number of Hidden Layers | Result |

0 - Only capable of representing linear separable functions or decisions.

1 - Can approximate any function that contains a continuous mapping from one finite space to another.

2 - Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.

I am familiar with the universal approximation theorem for 1 hidden layer, but not with the purported result about the additional power of 2 hidden layers. Is it true? If so, where can I find a detailed explanation and proof?

Edit: Apparently the table comes from Jeff Heaton.

• Perhaps not that important, but I think the key intuition would be the composition of non-linear activation functions. With no hidden layers, you have one activation function on the output. With one hidden layer, you now have one "internal" non-linear activation function and one after your output node. (Assuming a regression setting here.) With two hidden layers you now have an internal "composition" (may be misusing the term here) of two non-linear activation functions. – Wayne Nov 19 '17 at 17:43

It is by now well-known that functions computable by nets with a single hidden layer can approximate continuous functions, uniformly on compacts, under only weak assumptions on $\theta$. Consider now the following inversion problem: Given a continuous function $f : \mathbb{R}^m \rightarrow \mathbb{R}^p$, a compact subset $C \subseteq \mathbb{R}^p$ included in the image of $f$, and an $\varepsilon > 0$, find a function $\phi : \mathbb{R}^p \rightarrow \mathbb{R}^m$ so that $\|f(\phi(x)) - x \| < \varepsilon$ for all $x \in C$. It is trivial to see that in general discontinuous functions $\phi$ are needed. We show later that nets with just one hidden layer are not enough to guarantee the solution of all such problems, but nets with two hidden layers are.