In Goodfellow's Deep Learning book (http://www.deeplearningbook.org/contents/regularization.html 7.12) they state:

Because we usually use an inclusion probability of 1/2, the weight scaling rule usually amounts to dividing the weights by 2 at the end of training, and then using the model as usual. Another way to achieve the same result is to multiply the states of the units by 2 during training.

Could someone explain the purpose of rescaling when using dropout? I am having trouble grasping what exactly this is correcting for.


Consider the case you have dropout with p probability where p is (0,1] ,

the expected value of an output feature is p*E(WT+x),as only p units are used, say if feature>=4 then class A else B, now for the same input if in test time you do not have any dropout the Expected value of the activation is: E(WT+x) as all units are used, thus to prevent the decision boundary from shifting you reweigh the weights by 1/p to keep the expected activation same at the final layer.

In short you are doing weighted average(and not the addition) of the exponential set of networks learnt with dropout.

  • $\begingroup$ It seems to me that if dropout is used only in the last layer (as it is usually the case for convnets with a fully-connected layer at the end), then the scaling is not needed, because it will not affect choosing the largest output of the model (for classification task). $\endgroup$ – MichaelSB May 5 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.