Explanation of Spikes in training loss vs. iterations with Adam Optimizer

Question

I am training a neural network using i) SGD and ii) Adam Optimizer. When using normal SGD, I get a smooth training loss vs. iteration curve as seen below (the red one). However, when I used the Adam Optimizer, the training loss curve has some spikes. What's the explanation of these spikes?

Model Details:

14 input nodes -> 2 hidden layers (100 -> 40 units) -> 4 output units

I am using default parameters for Adam beta_1 = 0.9, beta_2 = 0.999, epsilon = 1e-8 and a batch_size = 32.

i) With SGD ii) With Adam

Answer 1

The spikes are an unavoidable consequence of Mini-Batch Gradient Descent in Adam (batch_size=32). Some mini-batches have 'by chance' unlucky data for the optimization, inducing those spikes you see in your cost function using Adam. If you try stochastic gradient descent (same as using batch_size=1) you will see that there are even more spikes in the cost function. The same doesn´t happen in (Full) Batch GD because it uses all training data (i.e the batch size is equal to the cardinality of your training set) each optimization epoch. As in your first graphic the cost is monotonically decreasing smoothly it seems the title (i) With SGD) is wrong and you are using (Full) Batch Gradient Descent instead of SGD.

On his great Deep Learning course at Coursera, Andrew Ng explains in great details this using the image below:

Answer 2

I've spent insane amount of time debugging exploding gradients and similar behaviour. Your answer will be dependent on loss function, data, architecture etc. There's hundreds of reasons. I'll name a few.

• Loss-dependent. Loglikelihood-losses needs to be clipped, if not, it may evaluate near log(0) for bad predictions/outliers in dataset, causing exploding gradients. Most packages (torch,tensorflow etc) implements clipping per default for their losses.
• Outliers in dataset.
• BatchNorm with small batchsize and large epsilon $\epsilon$ (hyperparameter). With batchnorm as $y=(x-u)/(s+\epsilon)$, then with small $s$ and $\epsilon$ you can get high magnitudes of $y$
• Final batch in an epoch may be small if dataset undivisible by batchsize. In torch dataloader there's a flag drop_last. Small batchsize=high variance

Now to why you see it with Adam and not with SGD? Clearly you reached lower loss with Adam. As noted before, If 99.9% of dataset has optima at one point except some observation, this may be that observation screaming "NO" and jumping out from the local minima when randomly selected to a batch. If you see it every dataset_size//batch_size+1-steps, it's probably due to final batchsize being small. I bet you'll see SGD spike too if you let it reach lower loss.

Bonus: Your really fast decrease with momentum-optimizer (Adam) could mean that some layer (input layer? output layer?) is initialized way out of scale (to large/small weights).