7
$\begingroup$

My CNN training gives me weird validation accuracy result. When it comes to 2.5,3.5,4.5 epochs, the validation accuracy is higher (meaning only need to go over half of the batches and I can reach better accuracy. But, If I go over all batches (one epoch), the validation accuracy drops). I repeat this experiment several times with random subset of data and the result looks similar.

Anything wrong here? When the accuracy is fluctuating? Also, when half cycle of epoch give better accuracy?

I use adadelta to train my network

enter image description here

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Were you able to figure out what was causing the issue? $\endgroup$
    – pseudomonas
    Jun 5, 2018 at 8:45
  • 1
    $\begingroup$ @RockTheStar How big is your dataset? What kinds of data such as image, audio, video? Also what is output of your neural network? Did you try different batch size? $\endgroup$
    – Cloud Cho
    Jul 23, 2019 at 16:04

3 Answers 3

Reset to default
8
$\begingroup$

This is likely due to the ordering of your dataset. If there's many observations of the same class in a sequence the weights of the network will move too far in the direction of classifying this class.

A common cause is if you balance the classes in your dataset by resampling observations and appending them to the dataset. Shuffle your dataset - that should help you avoid the fluctuations in accuracy (and perhaps obtain a higher accuracy overall).

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment. $\endgroup$
    – gung - Reinstate Monica
    Jan 8, 2016 at 13:24
  • $\begingroup$ I believe it did provide an answer, but I've now expanded a bit on it :) $\endgroup$
    – pir
    Jan 8, 2016 at 13:42
  • $\begingroup$ I have tried to shuffle the data, but I think I occasionally got similar graph. $\endgroup$
    – RockTheStar
    Jan 8, 2016 at 18:50
2
$\begingroup$

I had the same issue in the past and found out that the learning rate usually is the cause of oscillation. Try lowering your learning rate or using learning rate decay and keep training until the curve converges.

Improve this answer
$\endgroup$
0
$\begingroup$

Probably the learning rate is too high. The system is overfitting almost immediately, as overall the accuracy is falling after the first epoch. If you want to find the sweet spot using early stopping you surely need a lower learning rate to extend your choice. In addition, as suggested in other answers, I would use a learning rate scheduling.

Moreover, you may have a look at the size of your gradients. Exploding gradients may cause this kind of oscillation. Than using gradient clipping may be useful

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.