Dropout: scaling the activation versus inverting the dropout When applying dropout in artificial neural networks, one needs to compensate for the fact that at training time a portion of the neurons were deactivated. To do so, there exist two common strategies:


*

*scaling the activation at test time

*inverting the dropout during the training phase


The two strategies are summarized in the slides below, taken from Standford CS231n: Convolutional Neural Networks for Visual Recognition.
Which strategy is preferable, and why?

Scaling the activation at test time:

Inverting the dropout during the training phase:

 A: Andrew made very good explanation in his Deep Learning course on this session Dropout Regularization:


*

*Inverted dropout is more common because it makes the testing much easier

*The purpose of the inverting is to assure that the Z value will not be impacted by the reduce of W. 


Say a3 = a3 / keep_prob at the last step of implementation: 
Z[4] = W[4] * a[3] + b[4] , the element size  of a[3] has been reduced by keep_prob from D3(a percentage of elements have been dropped out by D3), thus the value of Z[4] is also gonna be reduced, so to compensate this roughly we shall invert the change by dividing keep_prob to make sure the value of Z[4] will not be impacted.

A: "inverting the dropout during the training phase" should be preferable.
Theoretically if we see Bernoulli dropout as a method of adding noise to the network, it's better that the noise could have a zero mean. If we do the scaling at training time to cancel out the portion of deactivated units, the mean of the noise would be zero.
There are other types of dropout/noise methods came out later (e.g. Gaussian multiplicative dropout, Gaussian additive noise) that also possess a zero mean.
In terms of training and testing neural networks in practice, there a reason to prefer such implementation as well.
Say I want to compare the performance of two models with the same architecture, one is trained by dropout and one is not.  
If I "scale the activation at test time", then I'll need two different networks at test time. If I use the "inverted version" then I could just apply the same test network (code) to both sets of trained parameters.
A: Another advantage of doing the inverted dropout (besides not having to change the code at test time) is that during training one can get fancy and change the dropout rate dynamically.  This has been termed as "annealed" dropout.  Essentially the logic is that adding dropout "noise" towards the beginning of training helps to keep the optimization from getting stuck in a local minimum, while letting it decay to zero in the end results in a finer tuned network with better performance.
ANNEALED DROPOUT TRAINING OF DEEP NETWORKS
Modified Dropout for Training Neural Network
