I am training a neural network using i) SGD and ii) Adam Optimizer. When using normal SGD, I get a smooth training loss vs. iteration curve as seen below (the red one). However, when I used the Adam Optimizer, the training loss curve has some spikes. What's the explanation of these spikes?
14 input nodes -> 2 hidden layers (100 -> 40 units) -> 4 output units
I am using default parameters for Adam
beta_1 = 0.9,
beta_2 = 0.999,
epsilon = 1e-8 and a
batch_size = 32.