Why does the loss/accuracy fluctuate during the training? (Keras, LSTM)

Question

I use LSTM network in Keras. During the training, the loss fluctuates a lot, and I do not understand why that would happen.

Here is the NN I was using initially:

And here are the loss&accuracy during the training:

(Note that the accuracy actually does reach 100% eventually, but it takes around 800 epochs.)

I thought that these fluctuations occur because of Dropout layers / changes in the learning rate (I used rmsprop/adam), so I made a simpler model:

I also used SGD without momentum and decay. I have tried different values for lr but still got the same result.

sgd = optimizers.SGD(lr=0.001, momentum=0.0, decay=0.0, nesterov=False)

But I still got the same problem: loss was fluctuating instead of just decreasing. I have always thought that the loss is just suppose to gradually go down but here it does not seem to behave like that.

So:

1. Is it normal for the loss to fluctuate like that during the training? And why it would happen?

2. If not, why would this happen for the simple LSTM model with the lr parameter set to some really small value?

Thanks. (Please, note that I have checked similar questions here but it did not help me to resolve my issue.)

Upd.: loss for 1000+ epochs (no BatchNormalization layer, Keras' unmodifier RmsProp):

Upd. 2: For the final graph:

model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
history = model.fit(train_x, train_y, epochs = 1500)

Data: sequences of values of the current (from the sensors of a robot).

Target variables: the surface on which robot is operating (as a one-hot vector, 6 different categories).

Preprocessing:

1. changed the sampling frequency so the sequences are not too long (LSTM does not seem to learn otherwise);
2. cut the sequences in the smaller sequences (the same length for all of the smaller sequences: 100 timesteps each);
3. check that each of 6 classes has approximately the same number of examples in the training set.

No padding.

Shape of the training set (#sequences, #timesteps in a sequence, #features):

(98, 100, 1) 

Shape of the corresponding labels (as a one-hot vector for 6 categories):

(98, 6)

Layers:

The rest of the parameters (learning rate, batch size) are the same as the defaults in Keras:

keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)

batch_size: Integer or None. Number of samples per gradient update. If unspecified, it will default to 32.

Upd. 3: The loss for batch_size=4:

For batch_size=2 the LSTM did not seem to learn properly (loss fluctuates around the same value and does not decrease).

Upd. 4: To see if the problem is not just a bug in the code: I have made an artificial example (2 classes that are not difficult to classify: cos vs arccos). Loss and accuracy during the training for these examples:

Answer 1

There are several reasons that can cause fluctuations in training loss over epochs. The main one though is the fact that almost all neural nets are trained with different forms of stochastic gradient descent. This is why batch_size parameter exists which determines how many samples you want to use to make one update to the model parameters. If you use all the samples for each update, you should see it decreasing and finally reaching a limit. Note that there are other reasons for the loss having some stochastic behavior.

This explains why we see oscillations. But in your case, it is more that normal I would say. Looking at your code, I see two possible sources.

1. Large network, small dataset: It seems you are training a relatively large network with 200K+ parameters with a very small number of samples, ~100. To put this into perspective, you want to learn 200K parameters or find a good local minimum in a 200K-D space using only 100 samples. Thus, you might end up just wandering around rather than locking down on a good local minima. (The wandering is also due to the second reason below).

2. Very small batch_size. You use very small batch_size. So it's like you are trusting every small portion of the data points. Let's say within your data points, you have a mislabeled sample. This sample when combined with 2-3 even properly labeled samples, can result in an update which does not decrease the global loss, but increase it, or throw it away from a local minima. When the batch_size is larger, such effects would be reduced. Along with other reasons, it's good to have batch_size higher than some minimum. Having it too large would also make training go slow. Therefore, batch_size should be treated as a hyperparameter.

Answer 2

Your loss curve doesn't look so bad to me. It should definitely "fluctuate" up and down a bit, as long as the general trend is that it is going down - this makes sense.

Batch size will also play into how your network learns, so you might want to optimize that along with your learning rate. Also, I would plot the entire curve (until it reaches 100% accuracy/minimum loss). It sounds like you trained it for 800 epochs and are only showing the first 50 epochs - the whole curve will likely give a very different story.

Answer 3

The fluctuations are normal within certain limits and depend on the fact that you use a heuristic method but in your case they are excessive. Despite all the performance takes a definite direction and therefore the system works. From the graphs you have posted, the problem depends on your data so it's a difficult training. If you have already tried to change the learning rate try to change training algorithm. You would agree to test your data: first compute the Bayes error rate using a KNN (use the trick regression in case you need), in this way you can check whether the input data contain all the information you need. Then try the LSTM without the validation or dropout to verify that it has the ability to achieve the result for you necessary. If the training algorithm is not suitable you should have the same problems even without the validation or dropout. Just at the end adjust the training and the validation size to get the best result in the test set. Statistical learning theory is not a topic that can be talked about at one time, we must proceed step by step.