117
$\begingroup$

Is there any general guidelines on where to place dropout layers in a neural network?

$\endgroup$
4
  • 3
    $\begingroup$ Using dropout regularization randomly disables some portion of neurons in a hidden layer. In the Keras library, you can add dropout after any hidden layer, and you can specify a dropout rate, which determines the percentage of disabled neurons in the preceding layer. $\endgroup$
    – redress
    May 31 '17 at 4:12
  • $\begingroup$ I do not think the ordering of the activation function and dropout matter. To see this, consider that in either case no information is propagated from the node, or backpropogated for that matter. $\endgroup$ Aug 7 '20 at 8:37
  • $\begingroup$ Most of the answers discussed above explain the application of dropout for fully connected networks. I would like to talk more about the dropout application in convolutional neural networks. Dropout is used to improve the generalization performance of the model. Generalization is achieved by making the learning features independent and not heavily correlated. Natural images are highly correlated (the image is a spatial data structure). The feature maps in CNNs also exhibit a strong correlation. 1. pixel and its surrounding pixels. 2. Surrounding pixels across the feature maps. To avoid this is $\endgroup$ Nov 23 '20 at 20:27
  • 2
    $\begingroup$ Please check my updates, and any suggestions or corrections would be highly appreciated. $\endgroup$ Jul 24 '21 at 3:43
75
$\begingroup$

In the original paper that proposed dropout layers, by Hinton (2012), dropout (with p=0.5) was used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers. This became the most commonly used configuration.

More recent research has shown some value in applying dropout also to convolutional layers, although at much lower levels: p=0.1 or 0.2. Dropout was used after the activation function of each convolutional layer: CONV->RELU->DROP.

$\endgroup$
4
  • 2
    $\begingroup$ So should they be placed after all layers, or only the ones with a non-linear activation? E.g. given a 2D convolution with a relu activation followed by a max pooling layer, should the (2D) dropout layer go immediately after the convolution, or after the max pooling layer, or both, or does it not matter? $\endgroup$
    – z0r
    Sep 23 '18 at 11:10
  • 2
    $\begingroup$ I've updated the answer to clarify that in the work by Park et al., the dropout was applied after the RELU on each CONV layer. I do not believe they investigated the effect of adding dropout following max pooling layers. $\endgroup$
    – 4Oh4
    Oct 8 '18 at 12:44
  • $\begingroup$ It's worth noting that in the Hinton paper, on page 10 (1938), they write that using dropout on convolutional layers when testing against the Google Street View dataset reduced classification error. $\endgroup$
    – Miki P
    Jan 23 '19 at 16:14
  • $\begingroup$ so does dropout process the weights, the pre_activation, the activations...? $\endgroup$ Jul 23 '21 at 16:55
21
$\begingroup$

In front of every linear projections. Refer to Srivastava et al. (2014).

$\endgroup$
3
  • 10
    $\begingroup$ The other answers describe how to apply dropout, but this is the only response that answers the OP question of where to apply dropout. $\endgroup$
    – stormont
    Nov 17 '17 at 0:38
  • 1
    $\begingroup$ The paper referred to masks the inputs to a linear transformation prior to applying the transformation (see the set of 4 equations in Section 4). So I guess @jnhwkim meant "before". $\endgroup$
    – Ben
    Jun 8 '20 at 12:20
  • $\begingroup$ so does dropout process the weights, the pre_activation, the activations...? $\endgroup$ Jul 23 '21 at 16:55
9
$\begingroup$

The original paper proposed dropout layers that were used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers.

We must not use dropout layer after convolutional layer as we slide the filter over the width and height of the input image we produce a 2-dimensional activation map that gives the responses of that filter at every spatial position. So as dropout layer neutralizes (makes it zero) random neurons there are chances of loosing very important feature in an image in our training process.

$\endgroup$
6
  • 3
    $\begingroup$ As I understand, there are chances of loosing very important features if we use dropouts on convolutional layers. So, we don't loose important features if we use dropouts on fully connected layers? $\endgroup$
    – Scott
    May 8 '20 at 8:42
  • $\begingroup$ The whole purpose of dropout layers is to tackle the problem of over-fitting and to introduce generalization to the model. Hence it is advisable to keep dropout parameter near 0.5 in hidden layers. It basically depend on number of factors including size of your model and your training data. For further reference link $\endgroup$ May 26 '20 at 8:21
  • 5
    $\begingroup$ I'm aware of the purpose of using dropout. You said in your answer that we might loose important features if we use dropout on convolutional layers. So, my question is: if we use dropout on fully connected layers, we don't loose important features? $\endgroup$
    – Scott
    Jun 21 '20 at 11:14
  • $\begingroup$ @Sherzod Hi, my answer is yes, we do loose important feature, but at the time of training, when we want to avoid to overfitting it seems helpful to loose. Something to refer link $\endgroup$ Aug 20 '20 at 6:37
  • $\begingroup$ so does dropout process the weights, the pre_activation, the activations...? $\endgroup$ Jul 23 '21 at 16:56
4
$\begingroup$

Some people interpret the dropout enabled neural network as an approximation of Bayesian Neural Network. And we can see this problem from the Bayesian perspective or treat such networks as stochastic artificial neural networks.

Artificial neural network

An artificial neural network maps some inputs/features to the output/predictions, which can be simplified as the following process:
$l_0 = x, $
$l_i = nl_i(W_il_{i-1}+b_i)\hspace{1cm} \forall i \in [1, n],$
$y=l_n.$
where $nl_i$ represents the non-linear activation function in the ith layer.

Stochastic artificial neural networks

There are two methods to convert a traditional neural network into a stochastic artificial neural network, simulating multiple possible models $\theta$ with their corresponding probability $p(\theta)$ distribution: 1) give the network stochastic activation(depicted below on the left), 2) or stochastic weights/coefficients(on the right).
enter image description here

The Dropout model

In this awesome article: What My Deep Model Doesn't Know... Yarin Gal views it as a stochastic network:

enter image description here

Notice that the dropout mechanism applied on $W_1$ works on the X layer and the dropout mechanism applied on $W_2$ works on the $\sigma$ layer.

And the process (with n layers) can be formulated as this:
$l_0 = x, $
$z_{i,j} \sim \text{Bernouilli}(p_i)\hspace{1cm} \forall i \in [1, n],$
$l_i = nl_i((l_{i-1} \cdot \text{diag} (z_i))W_i +b_i)\hspace{1cm} \forall i \in [1, n],$
$y=l_n.$

where $l_{i-1} \cdot \text{diag} (z_i)$ means that we randomly zero out some elements of the input(preceding layer) with probability $1-p_i$.

TL;DR

Then normally we apply the dropout before the activation to dropout the input elements in the preceding layer. Here is an illustration of the dropout machanism.

References:

  1. Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users
  2. What My Deep Model Doesn't Know...
$\endgroup$
4
  • $\begingroup$ Where is GeneralAttn defined? $\endgroup$ Sep 28 '17 at 23:35
  • $\begingroup$ so does dropout process the weights, the pre_activation, the activations...? $\endgroup$ Jul 23 '21 at 16:58
  • $\begingroup$ @rafaelvalle I updated my answer, and please check. $\endgroup$ Jul 24 '21 at 3:39
  • $\begingroup$ @CharlieParker I updated my answer, and please check $\endgroup$ Jul 24 '21 at 3:39
2
$\begingroup$

You apply dropout after the non-linear activation function.

Sources for this:

$\endgroup$
1
  • $\begingroup$ so does dropout process the weights, the pre_activation, the activations...? $\endgroup$ Jul 23 '21 at 16:58
2
$\begingroup$

For transformers I think you should do it like this:

According to the original paper (https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) they say:

Residual Dropout We apply dropout [27] to the output of each sub-layer, before it is added to the sub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks. For the base model, we use a rate of P_drop = 0.1.

which makes me think they do the following:

assert SubLayer is FullyConnected or MultiHeadedSeltAttention (not the output of LN+Add)
x = SubLayer(x)
x = torch.nn.dropout(x, p=0.1)
x = nn.LayerNorm(x) + x

So right after the multiheaded attention or fully connected (before the LN+ADD) during the transformer blocks/stack. But for the input just before the input to the actual encoder stack - i.e. together with the table look up embeddings.

batch_token_seqs: list[list[tokens]] = tokenize(batch_of_tokens)
batch_embeddings: torch.Tensor = table_look_up(batch_tokens) * D**0.5 # note this usually outputs masks, one for the right shift no cheating another for padding. D**0.5 is there for completeness, paper mentions it but doesn't justify it. It's not the same as the MHA division.
batch_embeddings = batch_embeddings + pos_embedding
batch_embeddings = self.drop_out(batch_embeddings)
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.