# Divergence in Stochastic Gradient Descent

I am training on a training set of 1.6 billion examples. After about 30 million examples, the training loss starts increasing after reaching a low.

The examples in training set are ordered temporally. What should I do so that the algorithm doesn't diverge?

Things I am considering:

• Lowering learning rate
• line search
• early stopping

which one should I pursue?

• How big is each of your batches for the SGD? What you may see is your algorithm avoiding overfitting. SGD sometimes takes several passes over your full dataset. You report this after going through less than 1% of your dataset. Why are you surprised? :D – usεr11852 Nov 25 '15 at 6:10
• Right now the batch size is just 1. I was thinking that the loss always decreases or stays the same. But after the low I mentioned it started increasing PS: I am not using any library but coding in python. – Swapniel Nov 25 '15 at 7:14
• and takes just 1 pass – Swapniel Nov 25 '15 at 7:28
• Apologies but I am confused, do you mean that you train for $n$ parameters with a sample size of $1$? – usεr11852 Nov 25 '15 at 7:54
• I mean I take one example at a time while updating. When you asked batch size I thought number of examples considered for 1 update of parameters like taking gradient on loss of 100 examples at a time. Sorry I am pretty new to this. – Swapniel Nov 25 '15 at 8:00