In the Fast Dropout article by Wang and Manning, they talk about approximating the input to a hidden layer by a Gaussian with the same mean and variance as the inputs to the layer (see page 5). But they never discuss how the mean or the variance would be computed. Do they select a minibatch and take statistics over the various elements of the minibatch each time we move to the next layer?

For example, suppose we have a minibatch $(\mathbf{x}_i, y_i)_{1 \leq i \leq n}$. Suppose we've just moved to layer $5$, and $f(\mathbf{x})$ represents the input to layer $5$ given input $\mathbf{x}$ to layer $0$. Then in order to find the appropriate Gaussian, we look at the mean and the variance of $\{f(\mathbf{x}_i)\}_{1 \leq i\leq n}$ in order to find the right Gaussian? (Sort of like we do in batch normalization?)


The paper describes on page 5:

... we may again approximate their inputs as Gaussians and characterize their outputs by the output means and variances

These means and variances are computed per sample: Every sample in the batch has its own mean and variance representing the distribution approximating the distribution of its values under various dropout masks. That's what is "fast" about this method; instead of direct sampling of dropout masks (which requires many samples, and thus dropout usually slows down the training), you pass the whole approximative distribution through the network, simultaneously evaluating how would the sample be perturbed by any of the dropout masks (or rather, how the whole distribution of the sample values would behave).

So, instead of outputting "values" of transformed samples $f(\mathbf{x})$, units output means and variances thereof: $\mathbb{E}[f(\mathbf{x})]$ and $\mathrm{var}[f(\mathbf{x})]$. You can use these in the next layer as the means and variances.

In the first layer, you consider the values of $\mathbf{x}$ to represent the individual means and variances are set to 0.

Here you can find my keras implementation of fast dropout.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.