I've been getting good results on a similar regression problem recently by fitting a deep sigmoid neural network in a greedy, layer-wise fashion. This approach is a little bit like the "Cascade-correlation architecture" described by Fahlman & Lebiere (1990) PDF, but the difference is that I'm adding entire layers rather than single units. It seems to work better with more units in each layer.
Here's what I've been doing:
- Train a regression network to convergence with one sigmoid hidden layer.
- Insert a second sigmoid hidden layer between the first hidden layer and the output layer. Train this network to convergence.
- Insert a third sigmoid hidden layer between the second hidden layer and the output layer. Train this network to convergence.
And so on. At each step, all the parameters in the model are eligible for gradient updates, but the newly-added layer and the output layer are the ones that appear to undergo the most significant changes.
Trying to train these layers all in one go is a non-starter because of the vanishing gradient problem, but training the layers using this greedy method seems to work pretty well.
Adding more hidden layers progressively gives me a massive boost in accuracy for the problem I'm working on (note the log scale on the vertical (error) axis):