# Explanation of Spikes in training loss vs. iterations with Adam Optimizer

I am training a neural network using i) SGD and ii) Adam Optimizer. When using normal SGD, I get a smooth training loss vs. iteration curve as seen below (the red one). However, when I used the Adam Optimizer, the training loss curve has some spikes. What's the explanation of these spikes?

Model Details:

14 input nodes -> 2 hidden layers (100 -> 40 units) -> 4 output units

I am using default parameters for Adam beta_1 = 0.9, beta_2 = 0.999, epsilon = 1e-8 and a batch_size = 32.

i) With SGD ii) With Adam

• For future notice, lowering your initial learning rate can help eliminate spikes in Adam Jun 19, 2018 at 21:54

The spikes are an unavoidable consequence of Mini-Batch Gradient Descent in Adam (batch_size=32). Some mini-batches have 'by chance' unlucky data for the optimization, inducing those spikes you see in your cost function using Adam. If you try stochastic gradient descent (same as using batch_size=1) you will see that there are even more spikes in the cost function. The same doesn´t happen in (Full) Batch GD because it uses all training data (i.e the batch size is equal to the cardinality of your training set) each optimization epoch. As in your first graphic the cost is monotonically decreasing smoothly it seems the title (i) With SGD) is wrong and you are using (Full) Batch Gradient Descent instead of SGD.

On his great Deep Learning course at Coursera, Andrew Ng explains in great details this using the image below:

• 'SGD...uses all training data'-are you sure of this? This means weights are updated after all the data is fed forward, but this is called full-batch gd, not sgd. Stochastic implies minibatch
– Alex
Sep 21, 2017 at 10:07
• Thank you @Alex for pointing my mistake, I already fixed it and improved the answer with reference for further information. Sep 21, 2017 at 13:07
• @xboard -- No, I am using mini-batch gradient descent for first one. Sep 25, 2017 at 9:12
• This answer is misleading. It says that SGD implies batch_size=1, which might be true in some old textbooks, but is just plain wrong in modern practice. Everybody uses minibatches with SGD because GPUs. I agree that full batch gradient descent is smoother, but in modern practice most interesting datasets are too large for for full batch GD. Jan 15 at 0:38

I've spent insane amount of time debugging exploding gradients and similar behaviour. Your answer will be dependent on loss function, data, architecture etc. There's hundreds of reasons. I'll name a few.

• Loss-dependent. Loglikelihood-losses needs to be clipped, if not, it may evaluate near log(0) for bad predictions/outliers in dataset, causing exploding gradients. Most packages (torch,tensorflow etc) implements clipping per default for their losses.
• Outliers in dataset.
• BatchNorm with small batchsize and large epsilon $$\epsilon$$ (hyperparameter). With batchnorm as $$y=(x-u)/(s+\epsilon)$$, then with small $$s$$ and $$\epsilon$$ you can get high magnitudes of $$y$$
• Final batch in an epoch may be small if dataset undivisible by batchsize. In torch dataloader there's a flag drop_last. Small batchsize=high variance

Now to why you see it with Adam and not with SGD? Clearly you reached lower loss with Adam. As noted before, If 99.9% of dataset has optima at one point except some observation, this may be that observation screaming "NO" and jumping out from the local minima when randomly selected to a batch. If you see it every dataset_size//batch_size+1-steps, it's probably due to final batchsize being small. I bet you'll see SGD spike too if you let it reach lower loss.

Bonus: Your really fast decrease with momentum-optimizer (Adam) could mean that some layer (input layer? output layer?) is initialized way out of scale (to large/small weights).