I am working on training a deep neural network (with pre-training) on millions of data recently. However, I found out that the loss shows a form of periodic phenomenon (about 60000 steps for 1 epoch), as shown below: Also, its performance of external evaluation on the validation set was fluctuating, as shown below (higher is better):
My questions are:
- What kind of causes can result in such phenomenon?
- Then what can I do to improve my network?