# Distribution of gradients across dimensions for neural networks

Does the distribution of gradients for neural networks known to follow a particular distribution?

That is, suppose I've a model with $$N$$ parameters. Then, the (stochastic) gradient at some point is a $$N$$ dimensional vector $$(x_1, x_2, \dots, x_N)$$.

Can we say anything regarding how $$x$$ values are distributed over this vector?

• This is an interesting question. I don't really have an answer for this, but ensuring that the distribution of gradients stays the same (namely, stays near 1) is a key motivation of the batch normalization technique, which serves to reduce exploding/vanishing gradients. – tchainzzz Jan 6 at 4:39