I am working on training a deep neural network (with pre-training) on millions of data recently. However, I found out that the loss shows a form of periodic phenomenon (about 60000 steps for 1 epoch), as shown below: loss_vs_updates Also, its performance of external evaluation on the validation set was fluctuating, as shown below (higher is better): eval_vs_updates

My questions are:

  1. What kind of causes can result in such phenomenon?
  2. Then what can I do to improve my network?

closed as off-topic by Sycorax, kjetil b halvorsen, Peter Flom Jun 2 '18 at 12:07

  • This question does not appear to be about statistics within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ I'm voting to close this question as off-topic because there's not enough information about how the network is constructed to provide a useful answer. $\endgroup$ – Sycorax Jun 1 '18 at 21:19
  • $\begingroup$ Two common causes of this behavior are 1. not shuffling the data 2. setting the learning rate too high $\endgroup$ – Sycorax Jun 1 '18 at 21:19