0
$\begingroup$

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability p and not present with a probability 1-p. In the test phase, all units are present, but we multiply each of them with the probability.

Now, is it like, let we have 4 input units originally named a,b,c,d. In the training stage, after applying dropout, with a dropout rate of 0.5, we are left with units a and c. So, As in the test stage, all the units are present, so, is it like, we multiply each of the units with 0.5? Also, Is p defined for each of the units in the network, or for the entire Neural Network?

Also, In doing so, how is the result same for training and test stage?

$\endgroup$

marked as duplicate by Reinstate Monica neural-networks Aug 10 at 1:06

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

0
$\begingroup$

Your description is more or less correct. First thing to note though is that dropout is usually applied on an entire layer, rather than a neural network. Of course, you could apply dropout to all layers in your neural network.

Let's assume we are dealing with a fully connected network. If dropout is applied during training with a factor of 0.5, that means that only half of all activation functions will deliver an output. The idea is that by training a different mix of units during each step, you get a more robust model that is able to deal well with noise/variability. (You can also compare it to the effect of ensemble networks.)

Now during testing however, we would like to use the full knowledge of the network, and thus not drop any units. Doing so would mean that our network is no longer properly scaled: the expected value during testing no longer matches the expected value during training. A simple way to fix this is indeed to scale all the activation outputs of the layer by the value of dropout (p).

When p is 0.5, during testing, this means we have double the amount of outputs at the layer in question, so we rescale all those outputs by multiplying them with 0.5.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.