I have ~30,000 instances and 500 dimensions. My target variable is continuous, so I'm preforming regression with no (i.e. linear) activation at the output.

I'm unable to fit the data, as my training error is higher than my test set error with tree-based models.

I've tried several different architectures, and I always under-fit the data. Are there any simple heuristics I should try?


I've been getting good results on a similar regression problem recently by fitting a deep sigmoid neural network in a greedy, layer-wise fashion. This approach is a little bit like the "Cascade-correlation architecture" described by Fahlman & Lebiere (1990) PDF, but the difference is that I'm adding entire layers rather than single units. It seems to work better with more units in each layer.

Here's what I've been doing:

  • Train a regression network to convergence with one sigmoid hidden layer.
  • Insert a second sigmoid hidden layer between the first hidden layer and the output layer. Train this network to convergence.
  • Insert a third sigmoid hidden layer between the second hidden layer and the output layer. Train this network to convergence.

And so on. At each step, all the parameters in the model are eligible for gradient updates, but the newly-added layer and the output layer are the ones that appear to undergo the most significant changes.

Trying to train these layers all in one go is a non-starter because of the vanishing gradient problem, but training the layers using this greedy method seems to work pretty well.

Adding more hidden layers progressively gives me a massive boost in accuracy for the problem I'm working on (note the log scale on the vertical (error) axis):

greedy layerwise regression


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.