Where should I place dropout layers in a neural network?

Question

Is there any general guidelines on where to place dropout layers in a neural network?

Answer 1

In the original paper that proposed dropout layers, by Hinton (2012), dropout (with p=0.5) was used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers. This became the most commonly used configuration.

More recent research has shown some value in applying dropout also to convolutional layers, although at much lower levels: p=0.1 or 0.2. Dropout was used after the activation function of each convolutional layer: CONV->RELU->DROP.

Answer 2

In front of every linear projections. Refer to Srivastava et al. (2014).

Answer 3

The original paper proposed dropout layers that were used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers.

We must not use dropout layer after convolutional layer as we slide the filter over the width and height of the input image we produce a 2-dimensional activation map that gives the responses of that filter at every spatial position. So as dropout layer neutralizes (makes it zero) random neurons there are chances of loosing very important feature in an image in our training process.

Answer 4

Some people interpret the dropout enabled neural network as an approximation of Bayesian Neural Network. And we can see this problem from the Bayesian perspective or treat such networks as stochastic artificial neural networks.

Artificial neural network

An artificial neural network maps some inputs/features to the output/predictions, which can be simplified as the following process:
$l_0 = x, $
$l_i = nl_i(W_il_{i-1}+b_i)\hspace{1cm} \forall i \in [1, n],$
$y=l_n.$
where $nl_i$ represents the non-linear activation function in the ith layer.

Stochastic artificial neural networks

There are two methods to convert a traditional neural network into a stochastic artificial neural network, simulating multiple possible models $\theta$ with their corresponding probability $p(\theta)$ distribution: 1) give the network stochastic activation(depicted below on the left), 2) or stochastic weights/coefficients(on the right).

The Dropout model

In this awesome article: What My Deep Model Doesn't Know... Yarin Gal views it as a stochastic network:

Notice that the dropout mechanism applied on $W_1$ works on the X layer and the dropout mechanism applied on $W_2$ works on the $\sigma$ layer.

And the process (with n layers) can be formulated as this:
$l_0 = x, $
$z_{i,j} \sim \text{Bernouilli}(p_i)\hspace{1cm} \forall i \in [1, n],$
$l_i = nl_i((l_{i-1} \cdot \text{diag} (z_i))W_i +b_i)\hspace{1cm} \forall i \in [1, n],$
$y=l_n.$

where $l_{i-1} \cdot \text{diag} (z_i)$ means that we randomly zero out some elements of the input(preceding layer) with probability $1-p_i$.

TL;DR

Then normally we apply the dropout before the activation to dropout the input elements in the preceding layer. Here is an illustration of the dropout machanism.

References:

1. Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users
2. What My Deep Model Doesn't Know...

Answer 5

You apply dropout after the non-linear activation function.

Sources for this:

• https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf - Formula on page 1933 and diagram on the next page.
• https://sebastianraschka.com/faq/docs/dropout-activation.html
• https://pgaleone.eu/deep-learning/regularization/2017/01/10/anaysis-of-dropout/

Answer 6

For transformers I think you should do it like this:

According to the original paper (https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) they say:

Residual Dropout We apply dropout [27] to the output of each sub-layer, before it is added to the sub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks. For the base model, we use a rate of P_drop = 0.1.

which makes me think they do the following:

assert SubLayer is FullyConnected or MultiHeadedSeltAttention (not the output of LN+Add)
x = SubLayer(x)
x = torch.nn.dropout(x, p=0.1)
x = nn.LayerNorm(x) + x

So right after the multiheaded attention or fully connected (before the LN+ADD) during the transformer blocks/stack. But for the input just before the input to the actual encoder stack - i.e. together with the table look up embeddings.

batch_token_seqs: list[list[tokens]] = tokenize(batch_of_tokens)
batch_embeddings: torch.Tensor = table_look_up(batch_tokens) * D**0.5 # note this usually outputs masks, one for the right shift no cheating another for padding. D**0.5 is there for completeness, paper mentions it but doesn't justify it. It's not the same as the MHA division.
batch_embeddings = batch_embeddings + pos_embedding
batch_embeddings = self.drop_out(batch_embeddings)