0
$\begingroup$

I have a question related to this question which is also yet to be answered.

I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the weights learned in the pre-training step for finetuning with dropout. Should I still rescale the learned weights from pre-training in the initialization even if the denoising level is chosen to be the same as the dropout rate?

In short: I have a network like this: 1000-100-10-100-1000. I use pre-training: 1000-100-1000; 100-10-100. Denoising level 0.3. Learning W1, W2. The full network is initialized with W1-W2-W2.T-W1.T. In training this network using a dropout rate of 0.3. Would I still need to employ the technique of rescaling the weights from pre-training as in this paper? For me it is not clear.

Thank you.

$\endgroup$
0
$\begingroup$

Short answer: yes, you should scale up the weights at test time (or scale them down during training). See my answer to the other question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.