Is there any general guidelines on where to place dropout layers in a neural network?
In the original paper that proposed dropout layers, by Hinton (2012), dropout (with p=0.5) was used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers. This became the most commonly used configuration.
More recent research has shown some value in applying dropout also to convolutional layers, although at much lower levels: p=0.1 or 0.2. Dropout was used after the activation function of each convolutional layer: CONV->RELU->DROP.
In front of every linear projections. Refer to Srivastava et al. (2014).
We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.
If I am not wrong, you can add it after the non-linearity of every cell:
layer_1 = (1/(1+np.exp(-(np.dot(X,synapse_0))))) if(do_dropout): layer_1 *= np.random.binomial([np.ones((len(X),hidden_dim))],1-dropout_percent) * (1.0/(1-dropout_percent))
The first line is the activation function, and the last is adding the dropout to the result. Please refer to this blog. Hope this helps.
Or you can place it to the input embedding as in this snippet:
class BahdanauAttnDecoderRNN(nn.Module): def __init__(self, hidden_size, output_size, n_layers=1, dropout_p=0.1): super(AttnDecoderRNN, self).__init__() # Define parameters self.hidden_size = hidden_size self.output_size = output_size self.n_layers = n_layers self.dropout_p = dropout_p self.max_length = max_length # Define layers self.embedding = nn.Embedding(output_size, hidden_size) self.dropout = nn.Dropout(dropout_p) self.attn = GeneralAttn(hidden_size) self.gru = nn.GRU(hidden_size * 2, hidden_size, n_layers, dropout=dropout_p) self.out = nn.Linear(hidden_size, output_size) def forward(self, word_input, last_hidden, encoder_outputs): # Note that we will only be running forward for a single decoder time step, but will use all encoder outputs # Get the embedding of the current input word (last output word) word_embedded = self.embedding(word_input).view(1, 1, -1) # S=1 x B x N word_embedded = self.dropout(word_embedded) # Calculate attention weights and apply to encoder outputs attn_weights = self.attn(last_hidden[-1], encoder_outputs) context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) # B x 1 x N # Combine embedded input word and attended context, run through RNN rnn_input = torch.cat((word_embedded, context), 2) output, hidden = self.gru(rnn_input, last_hidden) # Final output layer output = output.squeeze(0) # B x N output = F.log_softmax(self.out(torch.cat((output, context), 1))) # Return final output, hidden state, and attention weights (for visualization) return output, hidden, attn_weights
Technically you can add the dropout layer at the ending of a block, for instance after the convolution or after the RNN encoding.
The original paper proposed dropout layers that were used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers.
We must not use dropout layer after convolutional layer as we slide the filter over the width and height of the input image we produce a 2-dimensional activation map that gives the responses of that filter at every spatial position. So as dropout layer neutralizes (makes it zero) random neurons there are chances of loosing very important feature in an image in our training process.