I have few doubts around vanishing/exploding gradients.
The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers sometimes receive a very small gradient update. Because of which initial layers learn slowly, which in turn hampers the training of the overall network.
One solution is to initialize the weights with zero mean ,unit variance and then scale the weights with (1/N) where N is the number of neurons in a particular layer. Here we are trying to make variance in all layers as same as possible. We can also do a Xavier/He initialization to address the issue.
Now my questions
- Suppose I am training a MLP with say few layers (deep network). I have data that I divided into batches b1,b2...bn to feed the MLP. Lets say I do Xavier/He weight initialization. next, I start training the network. Now after I do back propagation over batch b1 or for that matter after any batch, We update the weights , which is in some way is going to be proportional to gradients. There is a good chance that this might distort the weights distribution which will make weight distribution not Xavier/He any more . The weights can resemble some random distribution which we avoided in the first place. How do you guarantee the weights are always distributed according to Xavier/He initialization after backpropagation?
- To avoid vanishing/Exploding gradient problem our weights needs to be around 1 (i.e 0.99 or 1.001) some thing like this. please correct if I am wrong? In any stage of the learning this has to be the case. My question is if the above statement is true then aren't we restricting the weights and is learning actually taking place ?
- How do you fix exploding gradient problem ? One of the solution I saw was clip the gradient if the exceeds certain threshold. Now how do you pick a good threshold?