highly sporadic validation error during training with multilayer perceptron

Question

I'm encountering an issue where a classifier I'm developing reports validation errors during training that span a wide range of values without consistently decreasing over time. Unfortunately, I'm new to ML and related topics and can't seem to diagnose the problem as a result.

The base code of the classifier comes from the theano deep_learning tutorials, specifically from the multilayer perceptron tutorial: link

I have changed the code from the tutorial in two ways: (1) Altered the data being used (2) Altered the topography of the MLP to account for a different number of input and output nodes

Here is an image of typical results:

The data I'm using comes from a wine quality experiment: link

I would appreciate help understanding:

(1) Why this is happening (2) How I should go about fixing it

Thanks

Answer 1

The amount of wiggle in your loss is related to the size of your batch. If you compute the loss over a larger batch size you should see a smoothened loss function. This means that if your validation loss was calculated on a larger batchsize, you won't see the sporadicity that you observe.

Here is an excerpt from Stanford's CNN course notes

The amount of “wiggle” in the loss is related to the batch size. When the batch size is 1, the wiggle will be relatively high. When the batch size is the full dataset, the wiggle will be minimal because every gradient update should be improving the loss function monotonically (unless the learning rate is set too high).

You can also check out this link which gives some more insight on training loss: http://cs231n.github.io/neural-networks-3/

Answer 2

I was having the same issue due to lack of centering of column features, once I did that my validation error smoothed out.

Answer 3

The issue I was having was that my learning rate was too high. I reduced it from .01 to .001 and that seems to have fixed the issue.

Answer 4

As mentioned in another answer, lowering the learning rate until the validation error stabilizes has worked for me. Also normalizing/standardizing input data (e.g. using sci-kit preprocessing) prior to running the model should help too!