When is a second hidden layer needed in feed forward neural networks?

I'm using a feed forward neural network to approximate a function with 24 inputs, and 3 outputs. Most of the literature suggests that a single layer neural network with a sufficient number of hidden neurons will provide a good approximation for most problems, and that adding a second or third layer yields little benefit.

However I have optimized a single layer, and a multi-layer neural network and my multi-layer network is much better. For the single layer network I performed a sweep of 1 to 80 neurons, retraining the network each time, and plotting the performance. After about 30 neurons the performance converged. For the multi-layer network I used the genetic algorithm to select the number of neurons in the first and second layer. This resulted in a much better performance.

What I would like to know is why this happened, considering that most of the literature suggests that 1 layer is enough. Is there a particular type of problem that requires more than one layer? Does this suggest that the function being approximated is discontinuous, not well-defined, jagged (not smooth), or all/ a mix of the above? Or does it suggest something else, or nothing at all? I know that when used for classification a multi-layer neural network can classify data that is not linearly separable, but I'm using the network for function approximation.

From a theoretical point of view you can approximate almost any function with one layer neural network.

There are some examples where a two layer neural network can approximate with a final number of nodes functions that with a one layer neural network can be approximated only with an infinitive number of neurons.

Try to increase the number of nodes in the one layer neural network or try to train your on layer neural network with another algorithm like PSO. It is often easy to fall in a local minimum.

• Thanks for your answer! Increasing the number of nodes in the one layer neural network had no effect after 30 nodes, as the performance plateaued after this point. It was only after adding the second layer that I saw a further increase in performance. The function can be approximated with a finite number of neurons in the one layer neural network, it is just better once a second layer (with 18 neurons in the first layer, and 5 in the second) is used. – Blue7 May 23 '14 at 18:21
• I've also tried to train the neural networks with PSO and GA in the past, but I got extremely bad results, so I just gave up on using global optimizers for training. I'm just looking for some justification/explanation to why the two layer network increased the performance, when the literature seems to disagree that this would happen. I've yet to find a source that says adding layers increases performance. – Blue7 May 23 '14 at 18:26
• I need to check. Are you using Sigmoid as link function and independent training set and test set? – Donbeo May 23 '14 at 20:53
• I'm using sigmoid as the activation function, and yes, I've split the training data into 80% actual training data, and 20% validation data (which tells it when to stop training). I also have a completly seperate data set to test it. – Blue7 May 23 '14 at 20:57
• Can you back up your statement that "a two layer neural network can approximate with a final number of nodes functions that with a one layer neural network can be approximated only with an infinitive number of neurons"? Wouldn't that go against the Universal Approximation Theorem? Thanks! – Ben Sep 27 '17 at 19:25

Stanford Professor Andrew Ng gave some guidelines for selecting a neural network architecture in his Machine Learning class on Coursera. I don't see the specific lecture videos on YouTube, but the course is free so it's no cost to access them on Coursera's site. Here's a summary of the relevant material.

In lecture 9-7 Putting it all together, general guidelines are given on picking default values for your neural network architecture.

Number of input units: Dimension of features x(i)
Number of output units: Number of classes

Reasonable default is one hidden layer, or if > 1 hidden layer, have the same number of hidden units in every layer (usually the more the better, anywhere from about 1X to 4X the number of input units).

In lecture 10-7 Deciding what to do next revisited, Professor Ng goes in to more detail.

Small neural networks:

• fewer parameters
• more prone to underfitting
• computationally cheaper

Large neural networks:

• more parameters
• more prone to overfitting
• computationally more expensive
• use regularization to address overfitting

Number of hidden layers: Split your data into training, cross validation, and test sets, and train neural networks with 1, 2, and 3 hidden layers, then see which one has the lowest cross validation error to choose an architecture.