How to detect vanishing and exploding gradients from change in kernel weights distribution? Edit: I've reworked my question to generalize better and be more on-topic for cross-validated, and be mostly software implementation agnostic.
Can vanishing gradients be detected by the change in distribution (or lack thereof) of my convolution's kernel weights throughout the training epochs? And if so how?
For example, if only 25% of my kernel's weights ever change throughout the epochs, does that imply an issue with vanishing gradients?
Here are my histograms and distributions, is it possible to tell whether my model suffers from Vanishing gradients from these images? (some middle hidden layers omitted for brevity) Thanks in advance.




 A: When the gradient vanishes for a particular layer it means that the gradient of the update for that layer is very close to 0, or actually is 0. When the gradient is 0, this implies that the weights do not update. When the weights do not update, the distribution plots will be the same. So as a heuristic, if you see that the distribution plots remain the same, you know the update was small (possibly even 0).
However, there is a possibility that the plots could remain the same for another reason. One example is suppose that gradient of the update was 0 for each parameter except for 2 parameters, and those 2 parameters happened to switch values. Cleary, the histogram will be the same (because the data has the same values, just in a different order); however, the gradient of this update could be arbitrarily large or arbitrarily small.
In other words, observing an unchanging histogram of parameter values is a necessary but not sufficient criterion to establishing that the gradient has vanished.
