I have a network whose input size is 100 and output size 2. Only these layers. I applied a dropout with keep_prob of 0.8 and I tried to understand the outcome.

As expected, the dropout mask has around 17-23 zeros every time time I run it, however, almost all the weights are updated. According to the paper:

Forward and back-propagation for that training case are done only on this thinned network.

So I was expecting that around 80 of my weights will change in each step of the training, but in reality they are all changing (in the beginning around 90-95 change, and in the next iterations all of them change).

I don't know if this has to do with the way Dropout is implemented in Tensorflow. Does somebody know why is this happening?

This is the code I'm running to check it.

import numpy as np
import tensorflow as tf

# As input, 100 random numbers.
input_size = 100
output_size = 2

x = tf.placeholder(tf.float32,[None, input_size],name="input")
y = tf.placeholder(tf.float32,[None, output_size],name="labels")

with tf.variable_scope("dense1") as scope:
    W = tf.get_variable("W",shape=[input_size,output_size],initializer=tf.keras.initializers.he_uniform())
    b = tf.get_variable("b",initializer=tf.zeros([output_size]))
    dropped = tf.nn.dropout(x,0.8)
    dense = tf.matmul(dropped,W)+b

eval_pred = tf.nn.sigmoid(dense,name="prediction")

cost = tf.reduce_mean(tf.losses.absolute_difference(eval_pred,y))
train_step = tf.train.AdamOptimizer(learning_rate=0.01).minimize(cost)

# 20 epochs, batch size of 1
epochs = 20

with tf.Session() as sess:

    allWeights = []
    for i in range(epochs):

        x_raw = np.random.random((1,input_size))
        y_raw = np.random.random((1,output_size))
        [_,c,d,w]=sess.run([train_step,cost,dropped,W], feed_dict={x: x_raw, y: y_raw})
        #print("Epoch {0}/{1}. Loss: {2}".format(i+1,epochs,c))

        # Numbers will be around 20% of input_size (17-22)

print("Calculate the difference between W_i and W_{i-1}")
for wi in range(1,len(allWeights)):
    difference = allWeights[wi]-allWeights[wi-1]
    # I expect that there will be around 20 weights that won't be updated
    # so the difference between the current weight and the previous one
    # should be zero.
  • $\begingroup$ I wonder if this is because Adam updates all parameters of the model via its averaging effects. $\endgroup$ – Sycorax says Reinstate Monica Feb 5 '18 at 3:07
  • $\begingroup$ You are right. I changed it to GradientDescentOptimizer and it gives the expected results. $\endgroup$ – jmlipman Feb 5 '18 at 3:13
  • 4
    $\begingroup$ To whom it may concern: this question is on-topic. Even though the title and body of the question suggest that this is a question about software, my answer, OP's acceptance of my answer and OP's validation of my comment all demonstrate that the questions is actually about optimization, and optimization is on-topic on CV.SE. $\endgroup$ – Sycorax says Reinstate Monica Feb 5 '18 at 15:48

This is because you're using the Adam optimizer. The Adam optimizer is a kind of momentum optimizer (specifically, it tracks the first and second moments of the updates), so an update will still occur for all model parameters even though dropout is present.

  • $\begingroup$ So, does it make sense to use dropout and Adam? $\endgroup$ – Aleksei Petrenko Sep 3 '18 at 8:50
  • $\begingroup$ I don’t know of any examples where it’s expressly wrong to use dropout and Adam. $\endgroup$ – Sycorax says Reinstate Monica Sep 3 '18 at 13:20

Dropout: Dropout in Tensorflow is implemented slightly different than in the original paper: instead of scaling the weights by 1/(1-p) after updating the weights (where p is the dropout rate), the neuron outputs (e.g., the outputs from ReLUs) are scaled by 1/(1-p) during the forward and backward passes. In this manner, the weights do not have to be scaled after updating.

Stochastic Gradient Descent (SGD) optimizer: SGD operates by shuffling the training data for each epoch and updating the weights using the negative gradient multiplied by the learning rate. Note that it does not take into account any past weight updates.

Optimization methods with adaptive learning rate: Adagrad, Adadelta and RMSprop are three optimizers which use previous values of gradients to adjust the learning rate. Note that, as SGD, the weight update is still performed using only the current gradient multiplied by this variable learning rate; hence, for mini-batch of size 1, these methods would not update some of the weights when using dropout.

Optimization methods using momentum: Methods such as Nesterov accelerated gradient (NAG) and Adam present a momentum term, which takes into account previous gradients for the current update. Therefore, for mini-batch of size 1, these methods would not update some of the weights only during the first few data samples - after which, most likely, all the neurons would have been activated and every weight would have a history of non-zero gradients.

I would suggest verifying the weight updates for the first few training data samples for mini-batch size of 1. This would clarify the impact of these different optimization methods on the current weight update when using dropout. Note that for large mini-batch sizes the gradient is computed via the mean updates; therefore, even after the first mini-batch, most likely all the weight updates will be non-zero.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.