As you know at the training step according to the Dropout technique we eliminate every neuron with probability $p$.

The question is do we eliminate neurons on every training instance (or batch) (or something else)?

By eliminating I mean a neuron doesn't participate in the training step but it's weights are not zeroed, because it might become alive on the next instance (batch), is it correct?

I would appreciate any clarification.


We can train using batch dropout or per example dropout. For your second question you are correct as you do not want the weights to fit the data perfectly,you inject randomness preventing coadaptation of the weights.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.