problem with vanishing/exploding gradient problems solution

Question

I have few doubts around vanishing/exploding gradients.

The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers sometimes receive a very small gradient update. Because of which initial layers learn slowly, which in turn hampers the training of the overall network.

One solution is to initialize the weights with zero mean ,unit variance and then scale the weights with (1/N) where N is the number of neurons in a particular layer. Here we are trying to make variance in all layers as same as possible. We can also do a Xavier/He initialization to address the issue.

Now my questions

1. Suppose I am training a MLP with say few layers (deep network). I have data that I divided into batches b₁,b₂...b_n to feed the MLP. Lets say I do Xavier/He weight initialization. next, I start training the network. Now after I do back propagation over batch b₁ or for that matter after any batch, We update the weights , which is in some way is going to be proportional to gradients. There is a good chance that this might distort the weights distribution which will make weight distribution not Xavier/He any more . The weights can resemble some random distribution which we avoided in the first place. How do you guarantee the weights are always distributed according to Xavier/He initialization after backpropagation?
2. To avoid vanishing/Exploding gradient problem our weights needs to be around 1 (i.e 0.99 or 1.001) some thing like this. please correct if I am wrong? In any stage of the learning this has to be the case. My question is if the above statement is true then aren't we restricting the weights and is learning actually taking place ?
3. How do you fix exploding gradient problem ? One of the solution I saw was clip the gradient if the exceeds certain threshold. Now how do you pick a good threshold?

Answer 1

The weights can resemble some random distribution which we avoided in the first place. How do you guarantee the weights are always distributed according to Xavier/He initialization after backpropagation?

You don't. As long as the weights start in the right place, the rest of training usually goes pretty well in terms of the gradient dynamics.

To avoid vanishing/Exploding gradient problem our weights needs to be around 1 (i.e 0.99 or 1.001) some thing like this. please correct if I am wrong? In any stage of the learning this has to be the case.

Generally they're close to 0, not 1.

My question is if the above statement is true then aren't we restricting the weights and is learning actually taking place ?

Regularization (of which initialization is a type) restricts the model, but not too much that the model can't learn.

How do you fix exploding gradient problem ? One of the solution I saw was clip the gradient if the exceeds certain threshold.

Residual connections quite effectively mitigate exploding gradients. Clipping gradients also works.

Now how do you pick a good threshold?

1 usually works. Failing that, hyperparameter optimization.