Understanding dropout method: one mask per batch, or more?

Question

I was talking with someone I know about the dropout method, and I realized we had different conceptions of how it worked. My impression was that there is one mask sampled per minibatch. His impression was that for each datum in the minibatch, we sample a different mask. So if there are three training examples in our minibatch, we sample three different masks.

Here is a passage from Goodfellow

This seems to support his idea. But if that is true, how do we backpropagate? For each individual datum in the minibatch, we'd be backpropagating through a different network.

“What's wrong with that?” might be one question. But to my mind, the whole point of a minibatch is that we only need to backpropagate once per batch, on the function $\ell(x_1, y_1) + \ell(x_2, y_2) + \ell(x_3 + y_3)$, say, if there are three things in our minibatch. But here that wouldn't be so simple--we'd have to do three separate backpropagations and sum them, since they all correspond to different networks.

How does this work, in the usual way of doing dropout? One mask per batch? Or multiple?

I tried to find the answer online, but everything I found describes backprop for dropout in terms of a single example.

Answer 1

For each individual datum in the minibatch, we'd be backpropagating through a different network

Let's illustrate the dropout using three different scenarios(can be just treated as a simple logistic regression) as follows where the blue points are inputs $X$ (features, or the output of a layer) and red points are weights/parameters $W$.

This is an illustration of an input with batch size one:

And here is that of batch size 3:

In the mini-batch situation, the losses of all cases in a mini-batch sum to one loss, and then the gradient of each case is just added to update the weights in one go. The process of mini-batch is often accelerated by vectorization.

Now let's add the dropout to the weights:

You see that the dropout is applied to inputs rather than parameters. Or the dropout mechanism applied on $W$ works on the $X$/preceding layer.

So, it is just like zero out(multiply a mask with 1's and 0's with the same shape of the input/blue point matrix) some of the input features and the back propagation just works as that for the middle illustration here. For details of the mini-batch gradient descent please refer to this answer.

An application of this trick is SimCSE.