Extremely large spike in training loss that destroys training progress

Question

I am training a super-resolution neural network that takes a low-resolution image and predicts its high-resolution counterpart. However, once in a while (e.g. a few hours), the training loss will spike to an extremely large value (e.g. 10^12 times the average loss before spike):

Inspecting the prediction results after the spike showed that the training progress is basically destroyed and started over, with worse accuracy than even the uninitialized network.

Some details and things I have tried:

• Loss function: I am using L2 loss comparing the predicted images, so there shouldn't be any numerical instabilities. I also used an L2 loss comparing the Sobel gradient of the images.
• Model: I used batch norm in one of the model branches, but the batch size is 8 which isn't too small. The model also has residual blocks.
• Optimizer: I am using Adam with lr=0.0005, betas=(0.9, 0.999), eps=1e-08.
• All images in the dataset should all have RGB value within [0, 1], so there shouldn't be any outliers.
• The loss explosion happened halfway through an epoch, so it's not caused by the (possibly smaller) last batch in the dataset.
• I am using gradient clipping, it seemed to make the explosion a bit less frequent but didn't get rid of it.
• I took a look at the checkpoint weights before and after the spike, seems like the L2 norm of the weights had an increase (from about 70 to 110).
• Note that the loss itself also doesn't seem to decrease even before the spike; however visually the result seems to be improving. Not sure if that is a problem on its own.

Answer 1

The problem is solved, though I am not sure which of the following things I tried was the reason.

• I increased the eps parameter of the batch norm layers to about 0.1. This way, if there's too little variation in a minibatch, the values won't necessarily explode.
• I did residual scaling of 0.1 for my residual blocks, basically multiplying the residual by 0.1 before adding to the skip connection.
• I lowered the learning rate to 1e-4.

Answer 2

Lowering the learning rate is a good suggestion, but it decreased the spike intensity instead removing them.

What worked in my case was adding a few BatchNormalization layers which helped stabilise the loss and the training.

I was using the Adam optimiser, and had a batch size of 32 in my case. The last batch with potentially few examples is dropped during my training round so it's not something that could have caused it.