Accuracy unchanged while error decreases

Question

I am trying to train a keras model for recognising human motion, where the input are the extracted features, such as 2D position of face, torso, etc.

Recently, I have managed to train a model with high accuracy for subject-dependent case, where the derivatives of the original human motion were recognised correctly with >90% accuracy

Now, I am interested in the subject-independent case. I use the same data, only I make sure that the subject I am testing on does not appear in the training set.

Unfortunately, my previous model did not work as the losses of the training and validation sets were increasing and the accuracy was near the random choice. So, I decided to tinker with the model, adding dropouts and regularizations as well as changing the loss functions and optimizers available with the Keras API.

My initial model was stacked LSTMs, which worked for the subject-dependent case.

Currently, the model is still stacked LSTMs, only with added recurrent and input dropout to every layer of the stack plus dropout layer in between the LSTM layers.

LSTM(l1_l2(0.01) regularizer + dropout + recurrent dropout 0.5) -> Dropout(0.5) -> LSTM(l1_l2(0.01) regularizer + dropout + recurrent dropout 0.5) -> Dropout(0.5) -> LSTM(l1_l2(0.01) regularizer + dropout + recurrent dropout 0.5) -> Dense

with the 'mean_squared_error' or 'categorical_crossentropy' loss function and RMSprop optimizer (0.001 learning rate)

No matter which hyperparameters I choose, the training pattern is the same: decreasing loss for both training and validation, while the accuracy is stagnant for both training and validation after the first epoch

Example:

Epoch 1/500
1280/1292 [============================>.] - ETA: 0s - loss: 295.3031 - acc: 0.2195Epoch 00000: val_acc improved from -inf to 0.32955

1292/1292 [==============================] - 37s - loss: 293.5951 - acc: 0.2190 - val_loss: 91.2733 - val_acc: 0.3295

Epoch 2/500
1280/1292 [============================>.] - ETA: 0s - loss: 114.1788 - acc: 0.2484Epoch 00001: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 113.6036 - acc: 0.2477 - val_loss: 47.1054 - val_acc: 0.3295

Epoch 3/500
1280/1292 [============================>.] - ETA: 0s - loss: 50.0548 - acc: 0.2477Epoch 00002: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 49.8489 - acc: 0.2477 - val_loss: 27.3694 - val_acc: 0.3295

Epoch 4/500
1280/1292 [============================>.] - ETA: 0s - loss: 27.9562 - acc: 0.2477Epoch 00003: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 27.8671 - acc: 0.2477 - val_loss: 17.8739 - val_acc: 0.3295

Epoch 5/500
1280/1292 [============================>.] - ETA: 0s - loss: 18.5775 - acc: 0.2500Epoch 00004: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 18.5214 - acc: 0.2477 - val_loss: 11.6678 - val_acc: 0.3295

Epoch 6/500
1280/1292 [============================>.] - ETA: 0s - loss: 14.0851 - acc: 0.2469Epoch 00005: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 14.0414 - acc: 0.2477 - val_loss: 9.3495 - val_acc: 0.3295

Epoch 7/500
1280/1292 [============================>.] - ETA: 0s - loss: 12.4618 - acc: 0.2500Epoch 00006: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 12.4193 - acc: 0.2477 - val_loss: 7.5683 - val_acc: 0.3295

Epoch 8/500
1280/1292 [============================>.] - ETA: 0s - loss: 11.6259 - acc: 0.2469Epoch 00007: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 11.5798 - acc: 0.2477 - val_loss: 7.0470 - val_acc: 0.3295

Epoch 9/500
1280/1292 [============================>.] - ETA: 0s - loss: 11.0960 - acc: 0.2461Epoch 00008: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 11.0611 - acc: 0.2477 - val_loss: 5.6133 - val_acc: 0.3295

Epoch 10/500
1280/1292 [============================>.] - ETA: 0s - loss: 10.5272 - acc: 0.2500Epoch 00009: val_acc did not improve
1292/1292 [==============================] - 35s - loss: 10.4809 - acc: 0.2477 - val_loss: 6.6481 - val_acc: 0.3295

My question is: Why accuracies remain unchanged no matter which parameters are used even after training for 100 epochs?

Answer 1

I found the problem. I assumed that the shuffle flag in Sequential.fit(..) shuffles the training and validation sets. Unfortunately, the flag shuffles the training set, but not validation. By shuffling manually the validation set, the accuracy of the model is now improving over the epochs

Answer 2

I am not sure your comment should be the solution to your problem. During validation, you will not compute the gradient of the error you will obtain for the observations. In other words, when you validate, the value of your trained parameters are fixed and normally shuffling your validation should not change anything.

With regards to your question, while the loss you are using and the accuracy should be correlated (i.e, the accuracy increases as the loss decreases) it does not have to happen right away (or at all). It might be that your loss was so bad at first that it has to go down quite a bit to start being able to differentiate classes.

Answer 3

I had the same problem where loss is decreasing but accuracy isn't changing from 0.5, although the data is shuffled in training and validation phases.

Finally, I got the accuracy to increase by changing metrics from:

metrics=['accuracy']

To:

metrics=['categorical_accuracy']

If you are doing binary classification, consider using:

metrics=['binary_accuracy']

More details here.