# Over which set of elements should I perform norm clipping of gradients for backpropagation?

I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the individual layers and tensors within the network. Specifically, I am unsure about whether to calculate the norm over the set of all variables/tensors or over each layer separately, and why.

L2 normalisation of gradients is performed by the tf.clip_by_global_norm function in tensorflow, and it defines the global norm (by which the gradients are adjusted) as;

global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))


where t_list is the list of tensors and l2norm(t) is a function that computes the magnitude of the input vector t.

What I am uncertain about is whether to compute global_norm for all tensors in the network, or for each layer or weight/bias vector independently. In my particular example, should t_list be the list of all variables (ie. flattened set of all weight and bias derivatives of my entire network), or should it be performed for each vector of biases and weights (and for each layer) independently? And why is one preferable over the other?

This answer from Stack Overflow alludes to these two possibilities and suggests that clipping on the global norm is desirable as doing it on each parameter set individually would result in a different normalisation scale for each set:

Despite what seems to be popular, you probably want to clip the whole gradient by its global norm... Clipping each gradient matrix individually changes their relative scale but is also possible

But it doesn't really provide more context than this, so I am still not covinced I understand the trade-offs behind each approach.

What is the global norm?

It's just the norm over all gradients as if they were concatenated together to form one global vector.

So regarding that question, you have to compute global_norm for all gradient tensors in the network (they are contained in t_list).

As you already said, there are different approaches to whether clip by the global norm or local norm or directly by values: