Convolutional neural network: why would training accuacy and well as validation accuracy fluctuate wildly?

Question

I am training a convnet on a binary classification problem using medical images. I;m doing a preliminary evaluation of various shallow nets to get a sense of what the best hyperparameters are likely to be. My dataset is small - about 2500 images in each of the 2 classes. I'm getting wild fluctuations in training accuracy as well as validation accuracy. Most of the questions online discuss this in the context of validation accuracy. But what does in mean when training accuracy is also fluctuating - to give an example:

 Epoch 1/40
 11/11 [==============================] - 108s 10s/step - loss: 0.9150 - acc: 0.6183 - val_loss: 3.7535 - val_acc: 0.7344
 Epoch 2/40
 11/11 [==============================] - 144s 13s/step - loss: 1.1297 - acc: 0.5153 - val_loss: 11.2487 - val_acc: 0.2656
 Epoch 3/40 
 11/11 [==============================] - 101s 9s/step - loss: 1.3976 - acc: 0.1474 - val_loss: 2.3932 - val_acc: 0.8516
 Epoch 4/40
 11/11 [==============================] - 149s 14s/step - loss: 0.7517 - acc:   0.4707 - val_loss: 3.1836 - val_acc: 0.7266
 Epoch 5/40
 11/11 [==============================] - 101s 9s/step - loss: 0.7319 - acc: 0.4734 - val_loss: 0.8679 - val_acc: 0.8750

My intution was that the learning rate was too high and I was bouncing around a local minimum but I lowered the learning rate from 0.01 to 1e-3 and it had no effect. I also wondered about the possibility of the network throwing NaN variables but I have no experience of that or what to do about it. Incidentally I am using the Adam optimizer which seems to across the board result in better test accuracies. I should add though that by best shallow network for 20 Epoch reached 62% accuracy on a binary problem (not much good).

If someone could give me some intuition on this pattern of behavior it would be great - I am pretty sure this is a symptom of one thing....I think....

If it were lack of data then why would I not see a gradual overfit with a really bad validation accuracy?

Answer 1

OP's network not only exhibits large fluctuations in accuracy, but also large fluctuations in the loss. This is a classic symptom of using too large a learning rate, too small a batch size, or both.

Even though OP reduced the learning rate from $10^{-2}$ to $10^{-3}$, this smaller learning rate might not be a small enough learning rate for the problem. Finding a good configuration of neural network parameters is like lock picking, and involves a lot of trial and error; trying one thing probably isn't enough experiments to find a good configuration of model parameters. Some optimizers, like SGD, can be very picky about the learning rate used.

More information:

• What should I do when my neural network doesn't learn?
• What could be the reasons that making validation loss jumping up and down?

Answer 2

One scenario where you see fluctuating training accuracy and loss is when you print out mini-batch accuracy and mini-batch loss. If that is the case, then try printing out train accuracy and loss for the whole training set. Accuracy should be increasing and loss should be decreasing if the network is getting trained properly.