Does weights learned from pre-training using a Denoising Autoencoder needs rescaling when using dropout in complete NN?

Question

I have a question related to this question which is also yet to be answered.

I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the weights learned in the pre-training step for finetuning with dropout. Should I still rescale the learned weights from pre-training in the initialization even if the denoising level is chosen to be the same as the dropout rate?

In short: I have a network like this: 1000-100-10-100-1000. I use pre-training: 1000-100-1000; 100-10-100. Denoising level 0.3. Learning W1, W2. The full network is initialized with W1-W2-W2.T-W1.T. In training this network using a dropout rate of 0.3. Would I still need to employ the technique of rescaling the weights from pre-training as in this paper?

Thank you.

Answer 1

Short answer: yes, you should scale up the weights at test time (or scale them down during training). See my answer to the other question.