I use LSTM network in Keras. During the training, the loss fluctuates a lot, and I do not understand why that would happen.
(Note that the accuracy actually does reach 100% eventually, but it takes around 800 epochs.)
I also used SGD without momentum and decay. I have tried different values for
lr but still got the same result.
sgd = optimizers.SGD(lr=0.001, momentum=0.0, decay=0.0, nesterov=False)
But I still got the same problem: loss was fluctuating instead of just decreasing. I have always thought that the loss is just suppose to gradually go down but here it does not seem to behave like that.
Is it normal for the loss to fluctuate like that during the training? And why it would happen?
If not, why would this happen for the simple LSTM model with the
lrparameter set to some really small value?
Thanks. (Please, note that I have checked similar questions here but it did not help me to resolve my issue.)
Upd. 2: For the final graph:
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy']) history = model.fit(train_x, train_y, epochs = 1500)
Data: sequences of values of the current (from the sensors of a robot).
Target variables: the surface on which robot is operating (as a one-hot vector, 6 different categories).
- changed the sampling frequency so the sequences are not too long (LSTM does not seem to learn otherwise);
- cut the sequences in the smaller sequences (the same length for all of the smaller sequences: 100 timesteps each);
- check that each of 6 classes has approximately the same number of examples in the training set.
Shape of the training set (#sequences, #timesteps in a sequence, #features):
(98, 100, 1)
Shape of the corresponding labels (as a one-hot vector for 6 categories):
The rest of the parameters (learning rate, batch size) are the same as the defaults in Keras:
keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)
batch_size: Integer or None. Number of samples per gradient update. If unspecified, it will default to 32.
The loss for
batch_size=2 the LSTM did not seem to learn properly (loss fluctuates around the same value and does not decrease).
Upd. 4: To see if the problem is not just a bug in the code: I have made an artificial example (2 classes that are not difficult to classify: cos vs arccos). Loss and accuracy during the training for these examples: