0
$\begingroup$

I'm trying to understand the Dropout algorithm. In this paper, the authors say that the nodes are randomly switched off with probability $p$ for each "Training Example".

Does this literally mean that every row in your dataset has a different dropout layout? Or, can "Training Example" mean a training batch?

$\endgroup$

1 Answer 1

1
$\begingroup$

As detailed in this question and answer, a training example refers to a "row", not a "batch".

The paper you refer to actually says the nodes are randomly switched off for "each presentation of each training example". So each time a training example (row) is used (assuming you train for more than one epoch), a different set of nodes are dropped out. There's no permanent association between a row and a "dropout layout" - over the entire training process all the data is used to train the whole network.

$\endgroup$
3
  • $\begingroup$ Surely that would create issues with backpropagation? The original paper says dropout applies to a "Training Case", which is explicitly a mini-batch. Have the authors used the wrong terminology, or is "Training Example" often used in place of "Training Case"? $\endgroup$
    – Connor
    Commented Dec 14, 2022 at 9:44
  • 1
    $\begingroup$ See the answer to the question I linked - it explains why backpropagation works with dropout. $\endgroup$
    – Lynn
    Commented Dec 14, 2022 at 10:48
  • $\begingroup$ Okay, interesting, conceptually I always considered it like training a mini neural net using a batch of examples. But I guess having a different dropout for every example is taking that concept to its maximum extent. I presume either choice is mathematically fine to apply. Has anyone studied what affect choosing one option over the other has on your final model? $\endgroup$
    – Connor
    Commented Dec 14, 2022 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.