I am training a convnet on a binary classification problem using medical images. I;m doing a preliminary evaluation of various shallow nets to get a sense of what the best hyperparameters are likely to be. My dataset is small - about 2500 images in each of the 2 classes. I'm getting wild fluctuations in training accuracy as well as validation accuracy. Most of the questions online discuss this in the context of validation accuracy. But what does in mean when training accuracy is also fluctuating - to give an example:

 Epoch 1/40
 11/11 [==============================] - 108s 10s/step - loss: 0.9150 - acc: 0.6183 - val_loss: 3.7535 - val_acc: 0.7344
 Epoch 2/40
 11/11 [==============================] - 144s 13s/step - loss: 1.1297 - acc: 0.5153 - val_loss: 11.2487 - val_acc: 0.2656
 Epoch 3/40 
 11/11 [==============================] - 101s 9s/step - loss: 1.3976 - acc: 0.1474 - val_loss: 2.3932 - val_acc: 0.8516
 Epoch 4/40
 11/11 [==============================] - 149s 14s/step - loss: 0.7517 - acc:   0.4707 - val_loss: 3.1836 - val_acc: 0.7266
 Epoch 5/40
 11/11 [==============================] - 101s 9s/step - loss: 0.7319 - acc: 0.4734 - val_loss: 0.8679 - val_acc: 0.8750

My intution was that the learning rate was too high and I was bouncing around a local minimum but I lowered the learning rate from 0.01 to 1e-3 and it had no effect. I also wondered about the possibility of the network throwing NaN variables but I have no experience of that or what to do about it. Incidentally I am using the Adam optimizer which seems to across the board result in better test accuracies. I should add though that by best shallow network for 20 Epoch reached 62% accuracy on a binary problem (not much good).

If someone could give me some intuition on this pattern of behavior it would be great - I am pretty sure this is a symptom of one thing....I think....

If it were lack of data then why would I not see a gradual overfit with a really bad validation accuracy?

  • $\begingroup$ Did you wait for more epochs? It might indeed take about 5 epochs for the optimizer to find a way. $\endgroup$ – Daniel Möller Dec 7 '17 at 17:19
  • $\begingroup$ I'm trying with a longer training cycle now but would you expect these kinds of fluctuations over 10 or 20 epochs? If not, I guess I am looking for some kind of intuition as to why this is happening. $\endgroup$ – GhostRider Dec 7 '17 at 17:21
  • $\begingroup$ Wht kind of network are you using ? It could be that it's too deep or wide for your dataset. Try to reduce the number of parameters. I shall add that this question would be more appropriate on the Cross Validated SE and i'm voting for closing. $\endgroup$ – Lescurel Dec 7 '17 at 21:12
  • $\begingroup$ Something I have noticed (and I'll admit, this is a rookie error) but my classes are in order in datasets and I'm using hdf5 files. I haven't been shuffling the data at all. I would imagine that would cause the effects above....if all batches for a while are one class and then another. Correct me if I'm wrong. $\endgroup$ – GhostRider Dec 7 '17 at 21:18
  • 1
    $\begingroup$ Considering that your loss goes up between epochs, it sounds like this is due to small batch size. This isn't necessarily bad, but you can try to increase the batch size to see if it makes things smoother. $\endgroup$ – Alex R. Dec 8 '17 at 0:05

OP's network not only exhibits large fluctuations in accuracy, but also large fluctuations in the loss. This is a classic symptom of using too large a learning rate, too small a batch size, or both.

Even though OP reduced the learning rate from $10^{-2}$ to $10^{-3}$, this smaller learning rate might not be a small enough learning rate for the problem. Finding a good configuration of neural network parameters is like lock picking, and involves a lot of trial and error; trying one thing probably isn't enough experiments to find a good configuration of model parameters. Some optimizers, like SGD, can be very picky about the learning rate used.

More information:

| cite | improve this answer | |

One scenario where you see fluctuating training accuracy and loss is when you print out mini-batch accuracy and mini-batch loss. If that is the case, then try printing out train accuracy and loss for the whole training set. Accuracy should be increasing and loss should be decreasing if the network is getting trained properly.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.