3
$\begingroup$

I just started learning about neural networks and was wondering what a neural network with 2 hidden layers is able to express over a neural network with just 1 hidden layer (where number of neurons are limited). Specifically, I am trying to come up with an binary classification example with a decision boundary that can be expressed with 2 hidden layers but not one (assuming input is a point in the 2D space, ReLU activation for the hidden layers, and sigmoid activation for the output). The 1-layer network has 4 hidden neurons in its layer while the 2-layer network has 2 neurons each layer, so still 4 hidden neurons total. I've tried looking at concentric circles, enclosed shapes, and nonlinear boundaries but have no luck so far.

What is an example of a binary decision boundary that 2 hidden layers (2 neurons each layer) are able to capture but 1 hidden layer (4 neurons) isn't? Is this even possible?

Update: I edited my question to a limited number of neurons as that the universal approximation theorem does not apply.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
3
$\begingroup$

Neural networks with one hidden layer are universal approximators (given some constraints), so you won't find such a class of functions. See Does the universal approximation theorem apply to ReLu?

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I believe that universal approximation theorem only holds for a large amount of neurons in the 1 hidden layer. What about if the 1 hidden layer network had 4 neurons in the hidden layer and the 2 layer network had 2 neurons each layer (so still 4 hidden neurons total)? $\endgroup$
    – Regina Dea
    Commented Mar 12 at 17:00
  • $\begingroup$ @ReginaDea in general it's a good idea to ask a new question in this case, otherwise people could keep narrowing down their question indefinitely. $\endgroup$
    – Firebug
    Commented Mar 12 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.