Does the distribution of gradients for neural networks known to follow a particular distribution?

That is, suppose I've a model with $N$ parameters. Then, the (stochastic) gradient at some point is a $N$ dimensional vector $(x_1, x_2, \dots, x_N)$.

Can we say anything regarding how $x$ values are distributed over this vector?

  • $\begingroup$ This is an interesting question. I don't really have an answer for this, but ensuring that the distribution of gradients stays the same (namely, stays near 1) is a key motivation of the batch normalization technique, which serves to reduce exploding/vanishing gradients. $\endgroup$ – tchainzzz Jan 6 at 4:39

The exact answer is going to depend greatly on the type of network, the inputs, how it's trained....

For a simple way to see this:

  • If we're at a (local) optimum, the full gradient (across the entire training dataset) will be zero. In the interpolating regime common to modern neural networks, the individual gradient for each training point may even be exactly zero; depending on the loss function and how much you've trained / etc, it might instead be mean zero but approximately normal, etc.

  • At initialization, when we're very far from a solution, the gradient may be extremely similar for different datapoints (and very far from zero).

In some particular limits (network becoming infinitely wide, initialized near zero, trained via SGD, no batch normalization, ...), the "neural tangent kernel" regime has an answer here: activations are distributed according to a particular Gaussian process, and their derivatives are too. (When using square loss, this means the final result corresponds to kernel ridge regression / Gaussian process regression with a particular kernel.) See e.g.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.