# Training of a Neural Network

I am trying to train an Artificial Neural Network for classification. In the input layers, I have 402 neurons; the first 400 are binary, and the last two are floating points in the range -1 to 1. In the hidden layer I have 400 neurons, and in the output layer I have a single node which I want to represent values between -1 and 1.

I have tried to train this network using a vectorized implementation of back-propagation which I have found online (I have tried different implementations, and also implemented one myself). My problem is, that my network does not seem to learn a lot. If my learning rate is higher than around 0.0001, I get into trouble, and quickly goes into a local minimum, and with a lower learning rate the learning is (obviously) very very slow.

I can as much training data as possible, so this is not a problem, but of course time is limiting, so I would like to be able to train this network in a decent time.

Do you have any intuition about what might be wrong, or how much data is needed to train this network of around 160.000 weights?

If it is relevant, I can upload some of the data.

Edit: Due to the comment by Martin, here is some learning statistics for different number of hidden neurons: Google Docs Spreadsheet

Another thing which I have observed, is that for my dataset, a constant output of 0.3 will result in a SSE of around 160, so i definitely want to get below this SSE.

-
add comment

## 2 Answers

If time is a limiting factor, you could try reducing the number of hidden units substantially. In my experience, it's very rare to need this many hidden units. I would start with a small number (less than ten) of hidden units, and see if this gives adequate performance.

If you're more worried about local minima, perhaps it's worth trying adding a stochastic component to your algorithm, based on simulated annealing or stochastic gradient descent. This may slow things down a bit, but will prevent local minima from being such a problem.

-
add comment

Gradient descent is usually a bad optimization algorithm. Try one of the following:

• stochastic gradient descent
• Levenberg-Marquardt
• conjugate gradient

These algorithms are usually

1. faster and
2. achieve better results

E. g. the "two spirals" problem can be solved by gradient descent in >10,000 iterations and by Levenberg-Marquardt in < 300 iterations.

I think there is an advice about the training set size in the neural network FAQ but I can't find it at the moment. Usually it should be number of inputs * n, where n sufficiently large (OK, this probably won't help you).

-
add comment