I am trying to experiment with dropout in 2 layer NN on notMNIST dataset using TensorFlow (assignment 3 in Google Deep Learning Course on Udacity). But adding dropout causes fall in test accuracy and overfitting.

Here is graph description:

image_size = 28
num_labels = 10
batch_size = 128
hidden_units = 1024

graph = tf.Graph()
with graph.as_default():
    X = tf.placeholder(tf.float32, shape=(None, image_size * image_size))
    y = tf.placeholder(tf.float32, shape=(None, num_labels))

    w1 = tf.Variable(tf.truncated_normal([image_size * image_size, hidden_units]))
    b1 = tf.Variable(tf.zeros([hidden_units]))
    layer1 = tf.nn.relu(tf.matmul(X, w1) + b1)

    keep_prob = tf.placeholder(tf.float32, shape=())
    dropout1 = tf.nn.dropout(layer1, keep_prob)

    w2 = tf.Variable(tf.truncated_normal([hidden_units, num_labels]))
    b2 = tf.Variable(tf.zeros([num_labels]))
    layer2 = tf.matmul(dropout1, w2) + b2

    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=layer2, labels=y))
    prediction = tf.nn.softmax(layer2)

    optimizer_step = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

And code for testing different dropout rates:

num_steps = 3001

fig = plt.figure(figsize=(16, 4))

j = 1
for prob in np.arange(0.1, 1.1, 0.1):
    steps = []
    minibatch = []
    valid = []
    test = 0
    with tf.Session(graph=graph) as sess:

        for step in range(num_steps):
            offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
            batch_X = train_dataset[offset:(offset + batch_size), :]
            batch_y = train_labels[offset:(offset + batch_size), :]

            optimizer_step.run(feed_dict={X: batch_X, y: batch_y, keep_prob: prob})
            if step % 500 == 0:
                minibatch_predicted = sess.run(prediction, feed_dict={X: batch_X, keep_prob: 1.0})
                minibatch.append(accuracy(minibatch_predicted, batch_y))

                valid_predicted = sess.run(prediction, feed_dict={X: valid_dataset, keep_prob: 1.0})
                valid.append(accuracy(valid_predicted, valid_labels))

        test_predicted = sess.run(prediction, feed_dict={X: test_dataset, keep_prob: 1.0})
        test = accuracy(test_predicted, test_labels)

    ax = fig.add_subplot(2, 5, j)
    ax.set_title('Dropout {0:.1f}'.format(prob))

    ax.plot(steps, minibatch, 'r')
    ax.plot(steps, valid, 'b')
    ax.plot(steps, [test]*len(steps), 'cyan')
    j += 1


Take a look at charts generated. Red is for minibatch accuracy, blue for validation set accuracy and cyan line is the final test accuracy computed after training. All statistics except for test evaluation is taken at each epoch that is divisible by 500.

Accuracy charts

It can be noticed that for all dropout rates red line is higher than blue one that can be the cause of overfitting.
Moreover test accuracy is significantly smaller than one for L2 regularization added for each layer.

Here is test accuracy plot for each dropout value. Orange line is an accuracy of 93.14% for L2 reg.

Test accuracy chart

Are there any suggestions on improving accuracy with dropout only on hidden layer and for the reason why L2 wins on test?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.