Batch normalization on MNIST tutorial

Question

I tried to apply batch normalization to my network (several 1D-convolution layers and then a couple of fully-connected layers). Results were bad or no significant improvement so I tried on one of MNIST TensorFlow tutorials. It is named 06_modern_convnet.py.

Here I also didn't notice improvement and even have got a feeling that BN is not well compatible with dropout. For example for the variant below I once got 97% accuracy after 10 epochs, once - 60%. Usually in similar networks 98.5-99% accuracy is achieved, so 97% looks big fail.

So the question: why BN doesn't work here? Or it works but I don't see improvement because...? Or results are already too good and theirs improvement is a matter of luck?...

I changed a bit number of layers, made closer to other MNIST tutorial: 2 convolution and 2 fully-connected ones. Here is the code, one of variants

"""Tutorial on how to build a convnet w/ modern changes, e.g.
Batch Normalization, Leaky rectifiers, and strided convolution.

Parag K. Mital, Jan 2016.
"""
# %%
import tensorflow as tf
from libs.batch_norm import batch_norm
from libs.activations import lrelu
from libs.connections import conv2d, linear, setProcessPriorityLow
from libs.datasets import MNIST

import matplotlib
matplotlib.use('Qt4Agg')


def batch_norm_tf_layer(x, tfIsTraining, scope='bn', affine=True):
    print("layer")
    return tf.contrib.layers.batch_norm(x,
             center=True, scale=True,
             is_training=tfIsTraining,
             scope=scope)

def no_batch_norm(x, tfIsTraining, scope='bn', affine=True):
    print("no BN")
    return x

batch_norm_func = batch_norm
# batch_norm_func = batch_norm_tf_layer
# batch_norm_func = no_batch_norm

# %% Setup input to the network and true output label.  These are
# simply placeholders which we'll fill in later.
mnist = MNIST()
x = tf.placeholder(tf.float32, [None, 784])
y = tf.placeholder(tf.float32, [None, 10])

# %% We add a new type of placeholder to denote when we are training.
# This will be used to change the way we compute the network during
# training/testing.
is_training = tf.placeholder(tf.bool, name='is_training')

# %% We'll convert our MNIST vector data to a 4-D tensor:
# N x W x H x C
x_tensor = tf.reshape(x, [-1, 28, 28, 1])

# %% We'll use a new method called  batch normalization.
# This process attempts to "reduce internal covariate shift"
# which is a fancy way of saying that it will normalize updates for each
# batch using a smoothed version of the batch mean and variance
# The original paper proposes using this before any nonlinearities
h_1 = lrelu(batch_norm_func(conv2d(x_tensor, 32, name='conv1'),
                            is_training, scope='bn1'), name='lrelu1')
h_2 = lrelu(batch_norm_func(conv2d(h_1, 64, name='conv2'),
                            is_training, scope='bn2'), name='lrelu2')

keep_prob = tf.placeholder(tf.float32)
h_2_flat = tf.reshape(h_2, [-1, 64 * 7 * 7])
# Additional dropout here seems to improve results a lot
h_3 = linear(h_2_flat, 500, scope='linear1', activation=tf.nn.tanh)
h_3 = tf.nn.dropout(h_3, keep_prob)
h_4 = linear(h_3, 10, scope='linear2_variant3')

y_pred = tf.nn.softmax(h_4)

# %% Define loss/eval/training functions
cross_entropy = -tf.reduce_sum(y * tf.log(y_pred))
train_step = tf.train.AdamOptimizer(2e-4).minimize(cross_entropy)

correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))

# %% We now create a new session to actually perform the initialization the
# variables:
sess = tf.Session()
sess.run(tf.global_variables_initializer())

# %% We'll train in minibatches and report accuracy:
n_epochs = 50
batch_size = 200
batch_count = mnist.train.num_examples // batch_size
for epoch_i in range(n_epochs):
    for batch_i in range(batch_count):
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
        sess.run(train_step, feed_dict={
            x: batch_xs, y: batch_ys, is_training: True,
            keep_prob: 0.7})      # Dropout coefficient (1 - no dropout)

        if (epoch_i == 0 and batch_i < 50 and batch_i % 2 == 1) or \
           (batch_i % (batch_count / 5) == batch_count / 5 - 1):
            print("%d,%d: %.4f" %
                  (epoch_i, batch_i + 1,
                   sess.run(accuracy,
                           feed_dict={
                               x: mnist.validation.images,
                               y: mnist.validation.labels,
                               is_training: False,
                               keep_prob: 1.0
                           })))

Answer 1

I suspect that during training, dropout increases the variance of the activations/feature maps, which batch normalization compensates for. At test or validation time, dropout is "disabled", which suddenly lowers the variance, and screws up the batch normalization (which uses the variance as estimated by a weighted moving average during training).

In practice, dropout is not used very often anymore, and batch norm + weight decay + data augmentation serves as sufficient regularization in most cases.

edit: A very relevant paper which answers pretty much the same question: https://arxiv.org/abs/1801.05134