I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the individual layers and tensors within the network. Specifically, I am unsure about whether to calculate the norm over the set of all variables/tensors or over each layer separately, and why.
L2 normalisation of gradients is performed by the
tf.clip_by_global_norm function in
tensorflow, and it defines the global norm (by which the gradients are adjusted) as;
global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))
t_list is the list of tensors and
l2norm(t) is a function that computes the magnitude of the input vector
What I am uncertain about is whether to compute
global_norm for all tensors in the network, or for each layer or weight/bias vector independently. In my particular example, should
t_list be the list of all variables (ie. flattened set of all weight and bias derivatives of my entire network), or should it be performed for each vector of biases and weights (and for each layer) independently? And why is one preferable over the other?
This answer from Stack Overflow alludes to these two possibilities and suggests that clipping on the global norm is desirable as doing it on each parameter set individually would result in a different normalisation scale for each set:
Despite what seems to be popular, you probably want to clip the whole gradient by its global norm... Clipping each gradient matrix individually changes their relative scale but is also possible
But it doesn't really provide more context than this, so I am still not covinced I understand the trade-offs behind each approach.