3
$\begingroup$

I am trying to understand exactly how the dropout method typically works. I have been looking at the original Hinton paper but I can't seem to pull out this final detail.

If I understand correctly, the hidden dropout component randomly assigns the activations for a given layer to 0. So an activation matrix (e.g. 3 nodes) like this:

node1   node2   node3
0.543   0.453   0.985
0.758   0.535   0.786

would result in something like this (assume dropout_hidden = 0.5)

node1   node2   node3
0.543   0       0.985
0.758   0       0

Now with the visible_dropout is it the same approach applied to the input variables where different samples have different variables set to zero or is an entire variable for all samples set to zero? For example:

          var1   var2   var3   var4   var5   var6
sample1   0.444  0.547  0.876  0.245  0.016  0.168
sample2   0.554  0.875  0.222  0.423  0.876  0.187
sample3   0.668  0.888  0.975  0.111  0.324  0.007

Assuming visible_dropout = 0.5:

random masking?

          var1   var2   var3   var4   var5   var6
sample1   0      0      0.876  0      0.016  0
sample2   0.554  0      0      0.423  0.876  0
sample3   0      0.888  0.975  0      0.324  0.007

or random variable masking?

          var1   var2   var3   var4   var5   var6
sample1   0.444  0      0.876  0.245  0      0
sample2   0.554  0      0.222  0.423  0      0
sample3   0.668  0      0.975  0.111  0      0

I think it is the former (random masking) but I can't find a confirmation on this anywhere.

$\endgroup$
2
  • $\begingroup$ It seems that the paper didn't say that dropout for visible units should be different from dropout for hidden units, so it should be the former I guess $\endgroup$
    – dontloo
    Commented Apr 22, 2016 at 3:14
  • $\begingroup$ Dropout of a whole predictor column would be a bad idea if applied everywhere, and particularly bad for non-image data. I'd default to thinking the salt-and-pepper type masking was being described unless explicitly told otherwise. $\endgroup$ Commented Apr 25, 2016 at 13:04

1 Answer 1

1
+50
$\begingroup$

In page 2 of the paper you can read: "by also dropping out a random 20% of the pixels". Pixels are variables (or features) in CNN. So with this hint you might think that this is random variable masking. Also this make sense because this is what some other algorithms are successfully using to avoid overfitting, such as Random Forest.

However, I read some papers on the subjects and most of them refers to dropout as a per sample random selection of weights. So each time a new sample is shown to the network a subset of weights will be deactivated. Thus actually creating a lot of networks, each seeing only 1 instance. Also this post (https://www.quora.com/How-is-dropout-applied-to-mini-batches-in-dropout-neural-networks-with-stochastic-gradient-descent) confirm that even for mini-batch training, each samples in the mini-batch actually goes through a different thinned network (network with some weight set to zero).

So, to my understanding of the litterature on the topics I believe that dropout refers to random masking.

You can find a lot papers on the subjects that will confirm what I said but here is one for instance, https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.