Why does pre-training help avoid the vanishing gradient problem?

Question

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down through the network becomes ever more small.
In fact, the value of the logistic's derivative is at most $0.25$, and across many layers this upper bound is factored in many times.
So, why does pre-training help to avoid this effect? The derivative of logistic function is always at most $0.25$ and, in fine tuning phase, I always use back propagation with my even smaller derivative flowing in the net. Why the initial weights setting should change this behavior?

Answer 1

Your question touches on two topics:

1. Preprocessing of the data.

2. Initialization of weights. For this question there are already good answers: What are good initial weights in a neural network?.

As for the first question, I shall refer to the paper: LeCun et al., Efficient Backprop, section 4.3. It is explained in great detail, among other issues about training. Nevertheless, some practices have changed since then. For example, ReLus have replaced tanh and sigmoids.

Answer 2

I think it does not solve the vanishing gradient problem. The main difference between DBN and a fully-connected feed-forward neural net is that DBN uses a stack of pre-trained restricted Boltzmann machines to initialize the network’s weights. But the root of the vanishing gradient problem is not about the weight initialization but the activation function used in each neuron, although a good weight initialization sometimes could lead to faster convergence.