1
$\begingroup$

I have a question related to this question.

I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the weights learned in the pre-training step for finetuning with dropout. Should I still rescale the learned weights from pre-training in the initialization even if the denoising level is chosen to be the same as the dropout rate?

In detail: I have a network like this: 1000-100-10-100-1000. I use pre-training: 1000-100-1000; 100-10-100. Denoising level 0.3. Learning W1, W2. The full network is initialized with W1-W2-W2.T-W1.T. In training this network using a dropout rate of 0.3 would I still need to employ the technique of rescaling the weights from pre-training as in this paper?

$\endgroup$

1 Answer 1

0
$\begingroup$

Short answer: yes, you should scale up the weights at test time (or scale them down during training). See my answer to the other question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.