54
$\begingroup$

Is there any general guidelines on where to place dropout layers in a neural network?

$\endgroup$
  • 1
    $\begingroup$ Using dropout regularization randomly disables some portion of neurons in a hidden layer. In the Keras library, you can add dropout after any hidden layer, and you can specify a dropout rate, which determines the percentage of disabled neurons in the preceding layer. $\endgroup$ – redress May 31 '17 at 4:12
  • $\begingroup$ is karma farming permitted here? $\endgroup$ – redress May 31 '17 at 4:28
  • 4
    $\begingroup$ @redress who farmed, and how? $\endgroup$ – Franck Dernoncourt May 31 '17 at 4:55
  • $\begingroup$ Did you resolve this answer? $\endgroup$ – Blaszard Aug 12 '17 at 4:53
  • $\begingroup$ What types of neural networks? CNN, RNN, other? $\endgroup$ – Wayne Dec 5 '17 at 23:06
26
$\begingroup$

In the original paper that proposed dropout layers, by Hinton (2012), dropout (with p=0.5) was used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers. This became the most commonly used configuration.

More recent research has shown some value in applying dropout also to convolutional layers, although at much lower levels: p=0.1 or 0.2. Dropout was used after the activation function of each convolutional layer: CONV->RELU->DROP.

$\endgroup$
  • $\begingroup$ So should they be placed after all layers, or only the ones with a non-linear activation? E.g. given a 2D convolution with a relu activation followed by a max pooling layer, should the (2D) dropout layer go immediately after the convolution, or after the max pooling layer, or both, or does it not matter? $\endgroup$ – z0r Sep 23 '18 at 11:10
  • 1
    $\begingroup$ I've updated the answer to clarify that in the work by Park et al., the dropout was applied after the RELU on each CONV layer. I do not believe they investigated the effect of adding dropout following max pooling layers. $\endgroup$ – 4Oh4 Oct 8 '18 at 12:44
  • $\begingroup$ It's worth noting that in the Hinton paper, on page 10 (1938), they write that using dropout on convolutional layers when testing against the Google Street View dataset reduced classification error. $\endgroup$ – Miki P Jan 23 at 16:14
9
$\begingroup$

In front of every linear projections. Refer to Srivastava et al. (2014).

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • 3
    $\begingroup$ The other answers describe how to apply dropout, but this is the only response that answers the OP question of where to apply dropout. $\endgroup$ – stormont Nov 17 '17 at 0:38
3
$\begingroup$

If I am not wrong, you can add it after the non-linearity of every cell:

layer_1 = (1/(1+np.exp(-(np.dot(X,synapse_0)))))
if(do_dropout):
    layer_1 *= np.random.binomial([np.ones((len(X),hidden_dim))],1-dropout_percent)[0] * (1.0/(1-dropout_percent))

The first line is the activation function, and the last is adding the dropout to the result. Please refer to this blog. Hope this helps.

Or you can place it to the input embedding as in this snippet:

class BahdanauAttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, n_layers=1, dropout_p=0.1):
        super(AttnDecoderRNN, self).__init__()

        # Define parameters
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.n_layers = n_layers
        self.dropout_p = dropout_p
        self.max_length = max_length

        # Define layers
        self.embedding = nn.Embedding(output_size, hidden_size)
        self.dropout = nn.Dropout(dropout_p)
        self.attn = GeneralAttn(hidden_size)
        self.gru = nn.GRU(hidden_size * 2, hidden_size, n_layers, dropout=dropout_p)
        self.out = nn.Linear(hidden_size, output_size)

    def forward(self, word_input, last_hidden, encoder_outputs):
        # Note that we will only be running forward for a single decoder time step, but will use all encoder outputs

        # Get the embedding of the current input word (last output word)
        word_embedded = self.embedding(word_input).view(1, 1, -1) # S=1 x B x N
        word_embedded = self.dropout(word_embedded)

        # Calculate attention weights and apply to encoder outputs
        attn_weights = self.attn(last_hidden[-1], encoder_outputs)
        context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) # B x 1 x N

        # Combine embedded input word and attended context, run through RNN
        rnn_input = torch.cat((word_embedded, context), 2)
        output, hidden = self.gru(rnn_input, last_hidden)

        # Final output layer
        output = output.squeeze(0) # B x N
        output = F.log_softmax(self.out(torch.cat((output, context), 1)))

        # Return final output, hidden state, and attention weights (for visualization)
        return output, hidden, attn_weights

Source: https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb

Technically you can add the dropout layer at the ending of a block, for instance after the convolution or after the RNN encoding.

$\endgroup$
  • $\begingroup$ Where is GeneralAttn defined? $\endgroup$ – rafaelvalle Sep 28 '17 at 23:35
0
$\begingroup$

The original paper proposed dropout layers that were used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers.

We must not use dropout layer after convolutional layer as we slide the filter over the width and height of the input image we produce a 2-dimensional activation map that gives the responses of that filter at every spatial position. So as dropout layer neutralizes (makes it zero) random neurons there are chances of loosing very important feature in an image in our training process.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.