7
$\begingroup$

My CNN training gives me weird validation accuracy result. When it comes to 2.5,3.5,4.5 epochs, the validation accuracy is higher (meaning only need to go over half of the batches and I can reach better accuracy. But, If I go over all batches (one epoch), the validation accuracy drops). I repeat this experiment several times with random subset of data and the result looks similar.

Anything wrong here? When the accuracy is fluctuating? Also, when half cycle of epoch give better accuracy?

I use adadelta to train my network

enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ Were you able to figure out what was causing the issue? $\endgroup$ Commented Jun 5, 2018 at 8:45
  • 1
    $\begingroup$ @RockTheStar How big is your dataset? What kinds of data such as image, audio, video? Also what is output of your neural network? Did you try different batch size? $\endgroup$
    – Cloud Cho
    Commented Jul 23, 2019 at 16:04

3 Answers 3

8
$\begingroup$

This is likely due to the ordering of your dataset. If there's many observations of the same class in a sequence the weights of the network will move too far in the direction of classifying this class.

A common cause is if you balance the classes in your dataset by resampling observations and appending them to the dataset. Shuffle your dataset - that should help you avoid the fluctuations in accuracy (and perhaps obtain a higher accuracy overall).

$\endgroup$
3
  • $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment. $\endgroup$ Commented Jan 8, 2016 at 13:24
  • $\begingroup$ I believe it did provide an answer, but I've now expanded a bit on it :) $\endgroup$
    – pir
    Commented Jan 8, 2016 at 13:42
  • $\begingroup$ I have tried to shuffle the data, but I think I occasionally got similar graph. $\endgroup$ Commented Jan 8, 2016 at 18:50
2
$\begingroup$

I had the same issue in the past and found out that the learning rate usually is the cause of oscillation. Try lowering your learning rate or using learning rate decay and keep training until the curve converges.

$\endgroup$
0
$\begingroup$

Probably the learning rate is too high. The system is overfitting almost immediately, as overall the accuracy is falling after the first epoch. If you want to find the sweet spot using early stopping you surely need a lower learning rate to extend your choice. In addition, as suggested in other answers, I would use a learning rate scheduling.

Moreover, you may have a look at the size of your gradients. Exploding gradients may cause this kind of oscillation. Than using gradient clipping may be useful

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.