Over which set of elements should I perform norm clipping of gradients for backpropagation?

Question

I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the individual layers and tensors within the network. Specifically, I am unsure about whether to calculate the norm over the set of all variables/tensors or over each layer separately, and why.

L2 normalisation of gradients is performed by the tf.clip_by_global_norm function in tensorflow, and it defines the global norm (by which the gradients are adjusted) as;

global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))

where t_list is the list of tensors and l2norm(t) is a function that computes the magnitude of the input vector t.

What I am uncertain about is whether to compute global_norm for all tensors in the network, or for each layer or weight/bias vector independently. In my particular example, should t_list be the list of all variables (ie. flattened set of all weight and bias derivatives of my entire network), or should it be performed for each vector of biases and weights (and for each layer) independently? And why is one preferable over the other?

This answer from Stack Overflow alludes to these two possibilities and suggests that clipping on the global norm is desirable as doing it on each parameter set individually would result in a different normalisation scale for each set:

Despite what seems to be popular, you probably want to clip the whole gradient by its global norm... Clipping each gradient matrix individually changes their relative scale but is also possible

But it doesn't really provide more context than this, so I am still not covinced I understand the trade-offs behind each approach.

Answer 1

What is the global norm?

It's just the norm over all gradients as if they were concatenated together to form one global vector.

So regarding that question, you have to compute global_norm for all gradient tensors in the network (they are contained in t_list).

As you already said, there are different approaches to whether clip by the global norm or local norm or directly by values:

https://www.tensorflow.org/api_guides/python/train#Gradient_Clipping

As you can imagine, if you have very large gradient for one parameter-array but all others gradients are relatively moderate, than you would reduce your weight updating feedback for those parameters if you use global_norm as you clipping method, as the global_norm will be pretty high due to the outlier gradient.

Whereas, if you use local norm, you could restrict to only this outlier gradient which has these extremely huge values and let all other gradients be as they are.

In the end this is another hyperparameter you have to experiment with. What I like to do, is to observe the gradients, if ALL of them are large, I often go with global_norm else if I see a large gradient for only one or a few paramters, I prefer to clip only those gradients by looking at their norm-value and deciding what threshold could be appropriate (look when your network starts to output nan's and clip before that!)