I'm currently working on a regression problem, using neural networks to constrain parameters for a complex physical scenario. I am searching the hyperparameter space for the best model and have thus far found a 33% decrease in loss for 2 layers over 1 layer (searching over reasonable number of neurons given the training size, input dimension etc. & accounting for overfitting with dropout and early stopping).
Now, I am trying to justify the motivation for using multiple hidden layers seeing as the improvement is significant, but also considering the universal approximation theorem and the potential to overfit.
I have come across the following description from a previous question:
| Number of Hidden Layers | Result |
0 - Only capable of representing linear separable functions or decisions.
1 - Can approximate any function that contains a continuous mapping from one finite space to another.
2 - Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.
My question is between the difference between 1 and 2 above. Doesn't smoothness imply continuity? And, why would 1 layer be able to model a continuous function but not a smooth one?
Furthermore, are there any other justifications for multiple hidden layers for regression problems?
(direction to material is greatly appreciated! Its hard to sort through the masses online)