I was talking with someone I know about the dropout method, and I realized we had different conceptions of how it worked. My impression was that there is one mask sampled per minibatch. His impression was that for each datum in the minibatch, we sample a different mask. So if there are three training examples in our minibatch, we sample three different masks.

Here is a passage from Goodfellow

enter image description here

This seems to support his idea. But if that is true, how do we backpropagate? For each individual datum in the minibatch, we'd be backpropagating through a different network.

What's wrong with that? might be one question. But to my mind, the whole point of a minibatch is that we only need to backpropagate once per batch, on the function $\ell(x_1, y_1) + \ell(x_2, y_2) + \ell(x_3 + y_3)$, say, if there are three things in our minibatch. But here that wouldn't be so simple--we'd have to do three separate backpropagations and sum them, since they all correspond to different networks.

How does this work, in the usual way of doing dropout? One mask per batch? Or multiple?

I tried to find the answer online, but everything I found describes backprop for dropout in terms of a single example.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.