I'm using OpenANN to train a neural network with one hidden layer and a softmax output layer with cross entropy as the error function. For my application, the conjugate gradient algorithm seems to be a good performing algorithm compared to for example LBFGS.
The image below shows the training error and validation error for one configuration of the neural network (20 inputs, 4 hidden neurons, 12 outputs). Is it 'normal' for CG to have spikes in the performance during training, and if so, what is the explanation for the spikes? The spikes only appear for CG and not for LBFGS and LMA.