5
$\begingroup$

I am trying to train an Artificial Neural Network for classification. In the input layers, I have 402 neurons; the first 400 are binary, and the last two are floating points in the range -1 to 1. In the hidden layer I have 400 neurons, and in the output layer I have a single node which I want to represent values between -1 and 1.

I have tried to train this network using a vectorized implementation of back-propagation which I have found online (I have tried different implementations, and also implemented one myself). My problem is, that my network does not seem to learn a lot. If my learning rate is higher than around 0.0001, I get into trouble, and quickly goes into a local minimum, and with a lower learning rate the learning is (obviously) very very slow.

I can train as much data as possible, so this is not a problem, but of course time is limiting, so I would like to be able to train this network in a decent time.

Do you have any intuition about what might be wrong, or how much data is needed to train this network of around 160.000 weights?

If it is relevant, I can upload some of the data.

Due to the comment by Martin, here is some learning statistics for different number of hidden neurons: Google Docs Spreadsheet

Another thing which I have observed, is that for my dataset, a constant output of 0.3 will result in a SSE of around 160, so I definitely want to get below this SSE.

$\endgroup$
  • $\begingroup$ You speak of classification but I'm pretty sure that what you're describing is a case of regression. As already answered, you should start reducing the number of hidden neurons because your model is probably too complex and unable to generalise. $\endgroup$ – Digio Nov 21 '15 at 0:47
7
$\begingroup$

If time is a limiting factor, you could try reducing the number of hidden units substantially. In my experience, it's very rare to need this many hidden units. I would start with a small number (less than ten) of hidden units, and see if this gives adequate performance.

If you're more worried about local minima, perhaps it's worth trying adding a stochastic component to your algorithm, based on simulated annealing or stochastic gradient descent. This may slow things down a bit, but will prevent local minima from being such a problem.

$\endgroup$
3
$\begingroup$

Gradient descent is usually a bad optimization algorithm. Try one of the following:

  • stochastic gradient descent
  • Levenberg-Marquardt
  • conjugate gradient

These algorithms are usually

  1. faster and
  2. achieve better results

E. g. the "two spirals" problem can be solved by gradient descent in >10,000 iterations and by Levenberg-Marquardt in < 300 iterations.

I think there is an advice about the training set size in the neural network FAQ but I can't find it at the moment. Usually it should be number of inputs * n, where n sufficiently large (OK, this probably won't help you).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.