Convolutional neural network: why would training accuacy and well as validation accuracy fluctuate wildly?

I am training a convnet on a binary classification problem using medical images. I;m doing a preliminary evaluation of various shallow nets to get a sense of what the best hyperparameters are likely to be. My dataset is small - about 2500 images in each of the 2 classes. I'm getting wild fluctuations in training accuracy as well as validation accuracy. Most of the questions online discuss this in the context of validation accuracy. But what does in mean when training accuracy is also fluctuating - to give an example:

 Epoch 1/40
11/11 [==============================] - 108s 10s/step - loss: 0.9150 - acc: 0.6183 - val_loss: 3.7535 - val_acc: 0.7344
Epoch 2/40
11/11 [==============================] - 144s 13s/step - loss: 1.1297 - acc: 0.5153 - val_loss: 11.2487 - val_acc: 0.2656
Epoch 3/40
11/11 [==============================] - 101s 9s/step - loss: 1.3976 - acc: 0.1474 - val_loss: 2.3932 - val_acc: 0.8516
Epoch 4/40
11/11 [==============================] - 149s 14s/step - loss: 0.7517 - acc:   0.4707 - val_loss: 3.1836 - val_acc: 0.7266
Epoch 5/40
11/11 [==============================] - 101s 9s/step - loss: 0.7319 - acc: 0.4734 - val_loss: 0.8679 - val_acc: 0.8750


My intution was that the learning rate was too high and I was bouncing around a local minimum but I lowered the learning rate from 0.01 to 1e-3 and it had no effect. I also wondered about the possibility of the network throwing NaN variables but I have no experience of that or what to do about it. Incidentally I am using the Adam optimizer which seems to across the board result in better test accuracies. I should add though that by best shallow network for 20 Epoch reached 62% accuracy on a binary problem (not much good).

If someone could give me some intuition on this pattern of behavior it would be great - I am pretty sure this is a symptom of one thing....I think....

If it were lack of data then why would I not see a gradual overfit with a really bad validation accuracy?

• Did you wait for more epochs? It might indeed take about 5 epochs for the optimizer to find a way. – Daniel Möller Dec 7 '17 at 17:19
• I'm trying with a longer training cycle now but would you expect these kinds of fluctuations over 10 or 20 epochs? If not, I guess I am looking for some kind of intuition as to why this is happening. – GhostRider Dec 7 '17 at 17:21
• Wht kind of network are you using ? It could be that it's too deep or wide for your dataset. Try to reduce the number of parameters. I shall add that this question would be more appropriate on the Cross Validated SE and i'm voting for closing. – Lescurel Dec 7 '17 at 21:12
• Something I have noticed (and I'll admit, this is a rookie error) but my classes are in order in datasets and I'm using hdf5 files. I haven't been shuffling the data at all. I would imagine that would cause the effects above....if all batches for a while are one class and then another. Correct me if I'm wrong. – GhostRider Dec 7 '17 at 21:18
• Considering that your loss goes up between epochs, it sounds like this is due to small batch size. This isn't necessarily bad, but you can try to increase the batch size to see if it makes things smoother. – Alex R. Dec 8 '17 at 0:05

Even though OP reduced the learning rate from $$10^{-2}$$ to $$10^{-3}$$, this smaller learning rate might not be a small enough learning rate for the problem. Finding a good configuration of neural network parameters is like lock picking, and involves a lot of trial and error; trying one thing probably isn't enough experiments to find a good configuration of model parameters. Some optimizers, like SGD, can be very picky about the learning rate used.