Neural Networks - Difference between 1 and 2 layers?

Question

I'm currently working on a regression problem, using neural networks to constrain parameters for a complex physical scenario. I am searching the hyperparameter space for the best model and have thus far found a 33% decrease in loss for 2 layers over 1 layer (searching over reasonable number of neurons given the training size, input dimension etc. & accounting for overfitting with dropout and early stopping).

Now, I am trying to justify the motivation for using multiple hidden layers seeing as the improvement is significant, but also considering the universal approximation theorem and the potential to overfit.

I have come across the following description from a previous question:

| Number of Hidden Layers | Result |

0 - Only capable of representing linear separable functions or decisions.

1 - Can approximate any function that contains a continuous mapping from one finite space to another.

2 - Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.

My question is between the difference between 1 and 2 above. Doesn't smoothness imply continuity? And, why would 1 layer be able to model a continuous function but not a smooth one?

Furthermore, are there any other justifications for multiple hidden layers for regression problems?

(direction to material is greatly appreciated! Its hard to sort through the masses online)

Answer 1

Now, I am trying to justify the motivation for using multiple hidden layers seeing as the improvement is significant, but also considering the universal approximation theorem and the potential to overfit.

The theorem is based on the assumption that you have some arbitrary large number of neurons in the hidden layer. If you add another hidden layer, you need an exponentially lower number of units to get the same number of connections. Hence, you can approx have the same capacity with an exponentially smaller number of units if you use multiple layers.

I.e., consider the case of 1024 units in the hidden layer. You can get the same number of connections by using 10 layers with 2 units each (2^10=1024). However, you only have 2*10=20 parameters in that case. As a rule of thumb, the more parameter you have, the easier it is to overfit.

The downside of using multiple layers though are the vanishing and exploding gradient problems. A good sweet spot for multilayer perceptrons is usually 2 layers. 3 layers almost never really works for simple multi-layer perceptrons.