# Neural Networks - Difference between 1 and 2 layers?

I'm currently working on a regression problem, using neural networks to constrain parameters for a complex physical scenario. I am searching the hyperparameter space for the best model and have thus far found a 33% decrease in loss for 2 layers over 1 layer (searching over reasonable number of neurons given the training size, input dimension etc. & accounting for overfitting with dropout and early stopping).

Now, I am trying to justify the motivation for using multiple hidden layers seeing as the improvement is significant, but also considering the universal approximation theorem and the potential to overfit.

I have come across the following description from a previous question:

| Number of Hidden Layers | Result |

0 - Only capable of representing linear separable functions or decisions.

1 - Can approximate any function that contains a continuous mapping from one finite space to another.

2 - Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.

My question is between the difference between 1 and 2 above. Doesn't smoothness imply continuity? And, why would 1 layer be able to model a continuous function but not a smooth one?

Furthermore, are there any other justifications for multiple hidden layers for regression problems?

(direction to material is greatly appreciated! Its hard to sort through the masses online)

• That quote is pretty silly, at least poorly worded - "function that contains a continuous mapping from one finite space to another" is really poor choice of words - nobody's interested in continuity of functions between finite spaces, because any function between discrete spaces is continuous. The author probably meant something like interpolation (given a sequence of points, and corresponding values, there exists a function that achieves these values). – Jakub Bartczuk Mar 14 at 15:40