1
$\begingroup$

I'm currently working on a regression problem, using neural networks to constrain parameters for a complex physical scenario. I am searching the hyperparameter space for the best model and have thus far found a 33% decrease in loss for 2 layers over 1 layer (searching over reasonable number of neurons given the training size, input dimension etc. & accounting for overfitting with dropout and early stopping).

Now, I am trying to justify the motivation for using multiple hidden layers seeing as the improvement is significant, but also considering the universal approximation theorem and the potential to overfit.

I have come across the following description from a previous question:

| Number of Hidden Layers | Result |

0 - Only capable of representing linear separable functions or decisions.

1 - Can approximate any function that contains a continuous mapping from one finite space to another.

2 - Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.

My question is between the difference between 1 and 2 above. Doesn't smoothness imply continuity? And, why would 1 layer be able to model a continuous function but not a smooth one?

Furthermore, are there any other justifications for multiple hidden layers for regression problems?

(direction to material is greatly appreciated! Its hard to sort through the masses online)

$\endgroup$
  • 1
    $\begingroup$ That quote is pretty silly, at least poorly worded - "function that contains a continuous mapping from one finite space to another" is really poor choice of words - nobody's interested in continuity of functions between finite spaces, because any function between discrete spaces is continuous. The author probably meant something like interpolation (given a sequence of points, and corresponding values, there exists a function that achieves these values). $\endgroup$ – Jakub Bartczuk Mar 14 at 15:40
1
$\begingroup$

Now, I am trying to justify the motivation for using multiple hidden layers seeing as the improvement is significant, but also considering the universal approximation theorem and the potential to overfit.

The theorem is based on the assumption that you have some arbitrary large number of neurons in the hidden layer. If you add another hidden layer, you need an exponentially lower number of units to get the same number of connections. Hence, you can approx have the same capacity with an exponentially smaller number of units if you use multiple layers.

I.e., consider the case of 1024 units in the hidden layer. You can get the same number of connections by using 10 layers with 2 units each (2^10=1024). However, you only have 2*10=20 parameters in that case. As a rule of thumb, the more parameter you have, the easier it is to overfit.

The downside of using multiple layers though are the vanishing and exploding gradient problems. A good sweet spot for multilayer perceptrons is usually 2 layers. 3 layers almost never really works for simple multi-layer perceptrons.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.