I have a question related to this question which is also yet to be answered.
I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the weights learned in the pre-training step for finetuning with dropout. Should I still rescale the learned weights from pre-training in the initialization even if the denoising level is chosen to be the same as the dropout rate?
In short: I have a network like this: 1000-100-10-100-1000. I use pre-training: 1000-100-1000; 100-10-100. Denoising level 0.3. Learning W1, W2. The full network is initialized with W1-W2-W2.T-W1.T. In training this network using a dropout rate of 0.3. Would I still need to employ the technique of rescaling the weights from pre-training as in this paper?
Thank you.