When is a second hidden layer needed in feed forward neural networks? I'm using a feed forward neural network to approximate a function with 24 inputs, and 3 outputs. Most of the literature suggests that a single layer neural network with a sufficient number of hidden neurons will provide a good approximation for most problems, and that adding a second or third layer yields little benefit.
However I have optimized a single layer, and a multi-layer neural network and my multi-layer network is much better. For the single layer network I performed a sweep of 1 to 80 neurons, retraining the network each time, and plotting the performance. After about 30 neurons the performance converged. For the multi-layer network I used the genetic algorithm to select the number of neurons in the first and second layer. This resulted in a much better performance.
What I would like to know is why this happened, considering that most of the literature suggests that 1 layer is enough. Is there a particular type of problem that requires more than one layer? Does this suggest that the function being approximated is discontinuous, not well-defined, jagged (not smooth), or all/ a mix of the above? Or does it suggest something else, or nothing at all? I know that when used for classification a multi-layer neural network can classify data that is not linearly separable, but I'm using the network for function approximation. 
 A: From a theoretical point of view you can approximate almost any function with one layer neural network. 
There are some examples where a two layer neural network can approximate with a finite number of nodes functions that  with a one layer neural network can be approximated only with an infinite number of neurons.
Try to increase the number of nodes in the one  layer neural network or try to train your on layer neural network with another algorithm like PSO. It is often easy to fall in a local minimum.
A: Stanford Professor Andrew Ng gave some guidelines for selecting a neural network architecture in his Machine Learning class on Coursera. I don't see the specific lecture videos on YouTube, but the course is free so it's no cost to access them on Coursera's site.  Here's a summary of the relevant material.
In lecture 9-7 Putting it all together, general guidelines are given on picking default values for your neural network architecture.

Number of input units: Dimension of features x(i)
  Number of output units: Number of classes
Reasonable default is one hidden layer, or if > 1 hidden layer, have the same number of hidden units in every layer (usually the more the better, anywhere from about 1X to 4X the number of input units).

In lecture 10-7 Deciding what to do next revisited, Professor Ng goes in to more detail.

Small neural networks:
  
  
*
  
*fewer parameters
  
*more prone to underfitting
  
*computationally cheaper
  
  
  Large neural networks:
  
  
*
  
*more parameters
  
*more prone to overfitting
  
*computationally more expensive
  
*use regularization to address overfitting
  
  
  Number of hidden layers: Split your data into training, cross validation, and test sets, and train neural networks with 1, 2, and 3 hidden layers, then see which one has the lowest cross validation error to choose an architecture.

