I am training a convnet on a binary classification problem using medical images. I;m doing a preliminary evaluation of various shallow nets to get a sense of what the best hyperparameters are likely to be. My dataset is small - about 2500 images in each of the 2 classes. I'm getting wild fluctuations in training accuracy as well as validation accuracy. Most of the questions online discuss this in the context of validation accuracy. But what does in mean when training accuracy is also fluctuating - to give an example:
Epoch 1/40 11/11 [==============================] - 108s 10s/step - loss: 0.9150 - acc: 0.6183 - val_loss: 3.7535 - val_acc: 0.7344 Epoch 2/40 11/11 [==============================] - 144s 13s/step - loss: 1.1297 - acc: 0.5153 - val_loss: 11.2487 - val_acc: 0.2656 Epoch 3/40 11/11 [==============================] - 101s 9s/step - loss: 1.3976 - acc: 0.1474 - val_loss: 2.3932 - val_acc: 0.8516 Epoch 4/40 11/11 [==============================] - 149s 14s/step - loss: 0.7517 - acc: 0.4707 - val_loss: 3.1836 - val_acc: 0.7266 Epoch 5/40 11/11 [==============================] - 101s 9s/step - loss: 0.7319 - acc: 0.4734 - val_loss: 0.8679 - val_acc: 0.8750
My intution was that the learning rate was too high and I was bouncing around a local minimum but I lowered the learning rate from 0.01 to 1e-3 and it had no effect. I also wondered about the possibility of the network throwing NaN variables but I have no experience of that or what to do about it. Incidentally I am using the Adam optimizer which seems to across the board result in better test accuracies. I should add though that by best shallow network for 20 Epoch reached 62% accuracy on a binary problem (not much good).
If someone could give me some intuition on this pattern of behavior it would be great - I am pretty sure this is a symptom of one thing....I think....
If it were lack of data then why would I not see a gradual overfit with a really bad validation accuracy?