Neural Network: Spikes using conjugate gradient

Question

I'm using OpenANN to train a neural network with one hidden layer and a softmax output layer with cross entropy as the error function. For my application, the conjugate gradient algorithm seems to be a good performing algorithm compared to for example LBFGS.

The image below shows the training error and validation error for one configuration of the neural network (20 inputs, 4 hidden neurons, 12 outputs). Is it 'normal' for CG to have spikes in the performance during training, and if so, what is the explanation for the spikes? The spikes only appear for CG and not for LBFGS and LMA.

Answer 1

This behavior is typical for the implementation of conjugate gradient (CG) in OpenANN. It is provided by the library ALGLIB. I don't know if the spikes are so striking in other CG implementations.CG does a linesearch to find a minimum error value in the negative (conjugate) gradient direction. I think the spikes occur during this phase. That does not mean that these are considered to be valid solutions.

I am curious what other people can say that have a better understanding of nonlinear conjugate gradient optimization.

Answer 2

To me it looks like you have an very imbalanced dataset, meaning your ratio between positive and negative examples (if this would be a binary classification, which there isn't.. but you get the point) is very skewed.

If you are now also just training on a certain subset (e.g. sgd or mini-batch), your model just might learn that the "best solution" is to answer '0' all the time (if there are many more negative than positive examples). if - on a certain iteration - there suddenly is a positive example, it will also be classified negative and your cross-entropy error skyrockets.