4
$\begingroup$

I was talking with someone I know about the dropout method, and I realized we had different conceptions of how it worked. My impression was that there is one mask sampled per minibatch. His impression was that for each datum in the minibatch, we sample a different mask. So if there are three training examples in our minibatch, we sample three different masks.

Here is a passage from Goodfellow

screenshot of textbook page

This seems to support his idea. But if that is true, how do we backpropagate? For each individual datum in the minibatch, we'd be backpropagating through a different network.

“What's wrong with that?” might be one question. But to my mind, the whole point of a minibatch is that we only need to backpropagate once per batch, on the function $\ell(x_1, y_1) + \ell(x_2, y_2) + \ell(x_3 + y_3)$, say, if there are three things in our minibatch. But here that wouldn't be so simple--we'd have to do three separate backpropagations and sum them, since they all correspond to different networks.

How does this work, in the usual way of doing dropout? One mask per batch? Or multiple?

I tried to find the answer online, but everything I found describes backprop for dropout in terms of a single example.

$\endgroup$
1
  • 1
    $\begingroup$ They don’t correspond to different networks. It’s still the same circuit that you backpropagate through. The only difference is that in some places, you have a multiplication by 0 to backpropagate through. In others, you have a multiplication by 1 (which changes nothing). You can perform the backward pass in the normal way. It looks a tad inefficient, but hardware parallelism (“vectorized operations”) would make up for a lot of this. $\endgroup$ Aug 26, 2021 at 15:21

1 Answer 1

4
$\begingroup$

For each individual datum in the minibatch, we'd be backpropagating through a different network

Let's illustrate the dropout using three different scenarios(can be just treated as a simple logistic regression) as follows where the blue points are inputs $X$ (features, or the output of a layer) and red points are weights/parameters $W$.

This is an illustration of an input with batch size one:

enter image description here

And here is that of batch size 3:

enter image description here

In the mini-batch situation, the losses of all cases in a mini-batch sum to one loss, and then the gradient of each case is just added to update the weights in one go. The process of mini-batch is often accelerated by vectorization.

Now let's add the dropout to the weights:

enter image description here

You see that the dropout is applied to inputs rather than parameters. Or the dropout mechanism applied on $W$ works on the $X$/preceding layer.

So, it is just like zero out(multiply a mask with 1's and 0's with the same shape of the input/blue point matrix) some of the input features and the back propagation just works as that for the middle illustration here. For details of the mini-batch gradient descent please refer to this answer.

An application of this trick is SimCSE.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.