3
$\begingroup$

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down through the network becomes ever more small.
In fact, the value of the logistic's derivative is at most $0.25$, and across many layers this upper bound is factored in many times.
So, why does pre-training help to avoid this effect? The derivative of logistic function is always at most $0.25$ and, in fine tuning phase, I always use back propagation with my even smaller derivative flowing in the net. Why the initial weights setting should change this behavior?

$\endgroup$
3
$\begingroup$

Your question touches on two topics:

  1. Preprocessing of the data.

  2. Initialization of weights. For this question there are already good answers: What are good initial weights in a neural network?.

As for the first question, I shall refer to the paper: LeCun et al., Efficient Backprop, section 4.3. It is explained in great detail, among other issues about training. Nevertheless, some practices have changed since then. For example, ReLus have replaced tanh and sigmoids.

$\endgroup$
  • $\begingroup$ What definition of "pre-training" do you use? Maybe I am missing something, but it seems to me that your definition is different from the one in the answers to What is pre training a neural netwok (which I thought was the most common definition, but I am not an expert) $\endgroup$ – Oren Milman Sep 27 '18 at 13:47
0
$\begingroup$

I think it does not solve the vanishing gradient problem. The main difference between DBN and a fully-connected feed-forward neural net is that DBN uses a stack of pre-trained restricted Boltzmann machines to initialize the network’s weights. But the root of the vanishing gradient problem is not about the weight initialization but the activation function used in each neuron, although a good weight initialization sometimes could lead to faster convergence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.