How to increase accuracy of All-CNN C on CIFAR-10 test set [closed]

Question

I am trying to implement the paper Striving for Simplicity specifically the model All-CNN C on CIFAR-10 without data augmentation. This model is said to be able to reach close to 91% accuracy on test set for CIFAR-10.

It now is close to 86% on test set

EDIT 1: With both architectures VALID and SAME convolutions I have reached 87% on test set still not good enough...I am running out of ideas.

EDIT 2: I tried He initialization it does speed up a bit the learning but I got to 87.6% something which does not seem to be a statistically significant increase in the accuracy...

EDIT 3: Still stuck at the same point however I am now normalizing the regularization factor $\lambda$ by batch_size*number_of_classes to make it even as I am using tf.reduce_mean() also I interchanged the place of gcn and zca whitening I now normalize then whiten it gives me a little bit more than 88% I am on the right path !

EDIT 4: I also did put some loss on the biases which I did not do before. I am gonna try working a bit on preprocessing but I think I did pretty much all things possible.

EDIT 5: I thought about it and I think I may have made a mistake with the relative scale of $\lambda$ and the loss function. I now think that $\lambda=0.001$ described by the article may be for the full sum of "errors" (by error I mean KL distance between probability for one instance) divided by batch_size in order to be batch-independant so the loss is the mean error in the batch. however as I used the complete tf.reduce_mean() my sum is divided by an additional number_of_classes hence in my equation I should divide $\lambda$ by number_of_classes and not number_of_classes*batch_size.

I really struggled to make this thing work (see this for a detailed version of my consecutive trials). I think I am now at a stage where this question has become more suitable for CV instead of SO.
I computed mean of features and ZCA matrix of training set (flattening each image into vectors of features). I used them to whiten training and test sets. Then I contrast normalised it using Goodfellow factor (Goodfellow et al 2013). Exactly as specified in the article.

The treatments are done in my dataset class which is basically the one found in Deep MNIST tutorial but adapted for CIFAR-10 and adding the preprocessing. Hence the method .next_batchthat shuffles the set before feeding it.

Here is my maincode:

# -*- coding: utf-8 -*-
"""
Created on Thu Jan 14 13:06:44 2016


"""

#%%
import tensorflow as tf
import os
import numpy as np
import dataset_class
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import glob
from PIL import Image
from scipy.spatial.distance import pdist


def cifar_10_reshape(batch_arg):
    output=np.reshape(batch_arg,(10000,3,32,32)).transpose(0,2,3,1)
    return output

def unpickle(file):
    import cPickle
    fo=open(file,'rb')
    dict=cPickle.load(fo)
    fo.close()
    return dict




#Loading cifar-10 data and reshaping it to be batch_sizex32x32x3
batch1=unpickle('cifar-10-batches-py/data_batch_1')
batch2=unpickle('cifar-10-batches-py/data_batch_2')
batch3=unpickle('cifar-10-batches-py/data_batch_3')
batch4=unpickle('cifar-10-batches-py/data_batch_4')
batch5=unpickle('cifar-10-batches-py/data_batch_5')



batch1_data=cifar_10_reshape(batch1['data'])
batch2_data=cifar_10_reshape(batch2['data'])
batch3_data=cifar_10_reshape(batch3['data'])
batch4_data=cifar_10_reshape(batch4['data'])
batch5_data=cifar_10_reshape(batch5['data'])

batch1_labels=batch1['labels']
batch2_labels=batch2['labels']
batch3_labels=batch3['labels']
batch4_labels=batch4['labels']
batch5_labels=batch5['labels']

test_batch=unpickle('cifar-10-batches-py/test_batch')
test_images=cifar_10_reshape(test_batch['data'])
test_labels_data=test_batch['labels']


train_images=np.concatenate((batch1_data,batch2_data,batch3_data,batch4_data,batch5_data),axis=0)
train_labels_data=np.concatenate((batch1_labels,batch2_labels,batch3_labels,batch4_labels,batch5_labels),axis=0)

#one-hot encodinf of labels
train_labels=np.zeros((50000,10),dtype=np.float32)
test_labels=np.zeros((10000,10),dtype=np.float32)

for i in range(50000):
    a=train_labels_data[i]
    train_labels[i,a]=1.

for j in range(10000):
    b=test_labels_data[j]
    test_labels[j,b]=1.









#Défining network
def weight_variable(shape):
        initial = tf.random_normal(shape, stddev=0.05)
        return tf.Variable(initial)
def bias_variable(shape):
        initial=tf.random_normal(0.05,shape=shape)
        return tf.Variable(initial)

def conv2dstride2(x,W):
        return tf.nn.conv2d(x,W,strides=[1, 2, 2, 1], padding='VALID')

def conv2d(x,W):
        return tf.nn.conv2d(x,W,strides=[1, 1, 1, 1], padding='VALID')
def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')



x=tf.placeholder(tf.float32, [None, None, None, 3],name="x-input")
y_=tf.placeholder(tf.float32, [None, 10],name="y-input")




keep_prob_input=tf.placeholder(tf.float32)
keep_prob_pool=tf.placeholder(tf.float32)


#First set of conv followed by conv stride 2 operation and dropout 0.5
W_conv1=weight_variable([3,3,3,96])
b_conv1=bias_variable([96])

x_dropped=tf.nn.dropout(x,keep_prob_input)
h_conv1=tf.nn.relu(conv2d(x_dropped,W_conv1)+b_conv1)


W_conv2=weight_variable([3,3,96,96])
b_conv2=bias_variable([96])

h_conv2=tf.nn.relu(conv2d(h_conv1,W_conv2)+b_conv2)



W_conv3=weight_variable([3,3,96,96])
b_conv3=bias_variable([96])


h_conv3=tf.nn.relu(conv2dstride2(h_conv2,W_conv3)+b_conv3)

h_conv3_dropped=tf.nn.dropout(h_conv3,keep_prob_pool)

#Second set of conv followed by conv stride 2 operation

W_conv4=weight_variable([3,3,96,192])
b_conv4=bias_variable([192])


h_conv4=tf.nn.relu(conv2d(h_conv3_dropped,W_conv4)+b_conv4)


W_conv5=weight_variable([3,3,192,192])
b_conv5=bias_variable([192])

h_conv5=tf.nn.relu(conv2d(h_conv4,W_conv5)+b_conv5)


W_conv7=weight_variable([3,3,192,192])
b_conv7=bias_variable([192])

h_conv7=tf.nn.relu(conv2dstride2(h_conv5,W_conv7)+b_conv7)

h_conv7_dropped=tf.nn.dropout(h_conv7,keep_prob_pool)

#Third set of conv followed by conv stride 2 operation

W_conv8=weight_variable([3,3,192,192])
b_conv8=bias_variable([192])

h_conv8=tf.nn.relu(conv2d(h_conv7_dropped,W_conv8)+b_conv8)

W_conv9=weight_variable([1,1,192,192])
b_conv9=bias_variable([192])

h_conv9=tf.nn.relu(conv2d(h_conv8,W_conv9)+b_conv9)

W_conv10=weight_variable([1,1,192,10])
b_conv10=bias_variable([10])

h_conv10=tf.nn.relu(conv2d(h_conv9,W_conv10)+b_conv10)

h_conv10_pooled=tf.nn.avg_pool(h_conv10,ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

y_conv10_reshaped=tf.reshape(h_conv10_pooled,(-1,10))

#y_conv=y_conv4_reshaped
y_conv=tf.nn.softmax(y_conv10_reshaped)

#Creating a decay for the learning rate at step 200, 250 and 300 (there is probably a better way of doing that)
NUM_EPOCHS_PER_DECAY_1=200
NUM_EPOCHS_PER_DECAY_2=250
NUM_EPOCHS_PER_DECAY_3=300
LEARNING_RATE_DECAY_FACTOR=0.1
num_batches_per_epoch=50000/128
decay_steps_1=int(num_batches_per_epoch*NUM_EPOCHS_PER_DECAY_1)
decay_steps_2=int(num_batches_per_epoch*NUM_EPOCHS_PER_DECAY_2)
decay_steps_3=int(num_batches_per_epoch*NUM_EPOCHS_PER_DECAY_3)
starter_learning_rate=0.05

global_step=tf.Variable(0, trainable=False)

decayed_learning_rate_1=tf.train.exponential_decay(starter_learning_rate, 
                                                 global_step, 
                                                 decay_steps_1, 
                                                 LEARNING_RATE_DECAY_FACTOR,
                                                 staircase=True)
decayed_learning_rate_2=tf.train.exponential_decay(decayed_learning_rate_1, 
                                                 global_step, 
                                                 decay_steps_2, 
                                                 LEARNING_RATE_DECAY_FACTOR,
                                                 staircase=True)
decayed_learning_rate=tf.train.exponential_decay(decayed_learning_rate_2, 
                                                 global_step, 
                                                 decay_steps_3, 
                                                 LEARNING_RATE_DECAY_FACTOR,
                                                 staircase=True)

saver=tf.train.Saver()

sess=tf.Session()







with tf.name_scope("xent") as scope:

    cross_entropy= -tf.reduce_mean(y_*tf.log(tf.clip_by_value(y_conv,10e-30,1.)))+\
    (0.001/10.)*(tf.nn.l2_loss(W_conv9)+tf.nn.l2_loss(W_conv10)+\
    tf.nn.l2_loss(W_conv8)+tf.nn.l2_loss(W_conv7)+\
    tf.nn.l2_loss(W_conv5)+\
    tf.nn.l2_loss(W_conv3)+tf.nn.l2_loss(W_conv4)+\
    tf.nn.l2_loss(W_conv2)+tf.nn.l2_loss(W_conv1)+tf.nn.l2_loss(biases))
    #the second term is the weight decay
    ce_summ=tf.scalar_summary("cross_entropy", cross_entropy)

#We are using stochastic gradient descent with fixed momentum of 0.9
with tf.name_scope("train") as scope:
    train_step=tf.train.MomentumOptimizer(decayed_learning_rate,0.9).minimize(cross_entropy,global_step)


with tf.name_scope("test") as scope:
    correct_prediction =tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))


    accuracy=tf.reduce_mean(tf.cast(correct_prediction,tf.float32),0)
    accuracy_summary=tf.scalar_summary("accuracy",accuracy)

with tf.name_scope("Learning_rate") as scope:

    tf.scalar_summary('learning_rate',decayed_learning_rate)





merged=tf.merge_all_summaries()
writer=tf.train.SummaryWriter("/cifar_logs",sess.graph_def)




#Actually training the network
init=tf.initialize_all_variables()
sess.run(init)
CIFAR=dataset_class.read_data_sets(train_images,train_labels,test_images,test_labels,0)


for i in range(150000):
    batch=CIFAR.train.next_batch(128)
    global_step=i
    if i % 1000 == 0:
        feed={x: batch[0], y_: batch[1], keep_prob_input: 1, keep_prob_pool: 1}
        result=sess.run([merged,accuracy],feed_dict=feed)
        summary_str=result[0]
        acc=result[1]
        print ("Accuracy at step %s is :%s" % (i,acc))
        writer.add_summary(summary_str, i)
    else:
        sess.run(train_step, feed_dict={x: batch[0], y_: batch[1], keep_prob_input: 0.8, keep_prob_pool: 0.5})


save_path= saver.save(sess,"/home/model.ckpt")   
print "Model saved in file: ", save_path


print "Final Accuracy on train set: "+str(sess.run(accuracy, feed_dict={x: CIFAR.train.images, y_: CIFAR.train.labels, keep_prob_input: 1, keep_prob_pool: 1})) 
print "Final Accuracy on test set: "+str(sess.run(accuracy, feed_dict={x: CIFAR.test.images[range(0,500)], y_: CIFAR.test.labels[range(0,500)], keep_prob_input: 1, keep_prob_pool: 1})) 

tf.Session.close(sess)

Do you see something that troubles you in my code ? I am disatisfied for exemple of my writing the weight decay in this ugly fashion. I used mean instead of sum for cross-entropy to avoid exploding ReLU grads. If you have some ideas to make the accuracy jump I am all ears.

Answer 1

I believe that your last pooling layer does not match the All-CNN article's architecture. They are doing a 6x6 avg pooling, it seems that your h_conv10_pooled is doing a 2x2 non-overlapping avg pooling.

A far less important point, but it is worth pointing out nonetheless: it seems that you have initialized all your weight layers using a Gaussian with standard deviation .05. I took a quick look at the All ConvNet article and they do not seem to list their weight initialization scheme. I also did not find any reference to "Glorot" or "He" (also referred to as "MSRA" or "MSR" for Microsoft Research Asia or Microsoft Research, respectively) which are very commonly used initialization schemes.

Unless I'm overlooking some small coding error, it may be that you are using a different initialization scheme than the original authors?

If fixing the pooling kernel doesn't work, you could email the authors and seeing if they could share the weight initialization scheme or try the aforementioned weight initialization schemes: Xavier Glorot and Yoshua Bengio's scheme in addition the He/MSR initialization that more recently came out; I believe both are readily available in TensorFlow. You may see some improved accuracy and some performance gains. I haven't tried benchmarking the All-CNN so I cannot be certain.

But if I had to bet the 6x6 avg pooling kernel may be the source of the inability to reproduce their experimental results.

Answer 2

I reimplemented all of it, now the accuracy on CIFAR-10 test set is at 89.31%.

The key points were :

-the preprocessing: GCN followed by ZCA-whitening in that order

-the mode of the convolutions: most of them are SAME convolutions (padding=1 for kernels of 3x3) except for the last two which are VALID, which leads to an 8x8xd output, which is then averaged.

-the initialization: small gaussian or He

-the scheduled decrease of learning rate: after 200, 250 and 300 epochs

-the size of the output the logits and softmax are of size 16 even if there are only 10 classes I guess it comes from the fact that with more logits softmax is more spread.

Nervana neon claims to have attained 89.5% on caffe and nervana but even if I would like it to match exactly this number I consider it to be close enough for my needs. Link towards my github.

Answer 3

I've trained an all-cnn-c model. On CIFAR-10, the test accuracy is 91.97% (there was a checkpoint with over 92%), and test loss is 0.4654. The architecture is almost the same as described in the paper, except that they didn't mention whether or not batch normalization was applied.

FYI:

Batch Normalization: After the ReLUs of all convolutional layers.

DropOut:

1. drop_rate=0.2 after input layer
2. drop_rate=0.5 after every convolutional layer with strides=2

Batch Size: 128

Initial Learning Rate: 0.01

Learning Rate Decay Scale: 0.1, applied after [200, 250, 300] epochs.

Initialization for convolutional layers: He Initialization.

Weight Decay: L2-Regularization with scale=0.001

Optimizer: momentum optimizer with momentum=0.9, use_nesterov=True

Data Augmentation: generate augmented data randomly using keras.preprocessing.image.ImageDataGenerator(zoom_range=[0.8,1.2], rotation_range=15, width_shift_range=.17, height_shift_range=.17, horizontal_flip=True)

Cross-Validation: Not in use.