# How to increase accuracy of All-CNN C on CIFAR-10 test set [closed]

I am trying to implement the paper Striving for Simplicity specifically the model All-CNN C on CIFAR-10 without data augmentation. This model is said to be able to reach close to 91% accuracy on test set for CIFAR-10.

It now is close to 86% on test set

EDIT 1: With both architectures VALID and SAME convolutions I have reached 87% on test set still not good enough...I am running out of ideas.

EDIT 2: I tried He initialization it does speed up a bit the learning but I got to 87.6% something which does not seem to be a statistically significant increase in the accuracy...

EDIT 3: Still stuck at the same point however I am now normalizing the regularization factor $\lambda$ by batch_size*number_of_classes to make it even as I am using tf.reduce_mean() also I interchanged the place of gcn and zca whitening I now normalize then whiten it gives me a little bit more than 88% I am on the right path !

EDIT 4: I also did put some loss on the biases which I did not do before. I am gonna try working a bit on preprocessing but I think I did pretty much all things possible.

EDIT 5: I thought about it and I think I may have made a mistake with the relative scale of $\lambda$ and the loss function. I now think that $\lambda=0.001$ described by the article may be for the full sum of "errors" (by error I mean KL distance between probability for one instance) divided by batch_size in order to be batch-independant so the loss is the mean error in the batch. however as I used the complete tf.reduce_mean() my sum is divided by an additional number_of_classes hence in my equation I should divide $\lambda$ by number_of_classes and not number_of_classes*batch_size.

I really struggled to make this thing work (see this for a detailed version of my consecutive trials). I think I am now at a stage where this question has become more suitable for CV instead of SO.
I computed mean of features and ZCA matrix of training set (flattening each image into vectors of features). I used them to whiten training and test sets. Then I contrast normalised it using Goodfellow factor (Goodfellow et al 2013). Exactly as specified in the article.

The treatments are done in my dataset class which is basically the one found in Deep MNIST tutorial but adapted for CIFAR-10 and adding the preprocessing. Hence the method .next_batchthat shuffles the set before feeding it.

Here is my maincode:

# -*- coding: utf-8 -*-
"""
Created on Thu Jan 14 13:06:44 2016

"""

#%%
import tensorflow as tf
import os
import numpy as np
import dataset_class
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import glob
from PIL import Image
from scipy.spatial.distance import pdist

def cifar_10_reshape(batch_arg):
output=np.reshape(batch_arg,(10000,3,32,32)).transpose(0,2,3,1)
return output

def unpickle(file):
import cPickle
fo=open(file,'rb')
fo.close()
return dict

batch1=unpickle('cifar-10-batches-py/data_batch_1')
batch2=unpickle('cifar-10-batches-py/data_batch_2')
batch3=unpickle('cifar-10-batches-py/data_batch_3')
batch4=unpickle('cifar-10-batches-py/data_batch_4')
batch5=unpickle('cifar-10-batches-py/data_batch_5')

batch1_data=cifar_10_reshape(batch1['data'])
batch2_data=cifar_10_reshape(batch2['data'])
batch3_data=cifar_10_reshape(batch3['data'])
batch4_data=cifar_10_reshape(batch4['data'])
batch5_data=cifar_10_reshape(batch5['data'])

batch1_labels=batch1['labels']
batch2_labels=batch2['labels']
batch3_labels=batch3['labels']
batch4_labels=batch4['labels']
batch5_labels=batch5['labels']

test_batch=unpickle('cifar-10-batches-py/test_batch')
test_images=cifar_10_reshape(test_batch['data'])
test_labels_data=test_batch['labels']

train_images=np.concatenate((batch1_data,batch2_data,batch3_data,batch4_data,batch5_data),axis=0)
train_labels_data=np.concatenate((batch1_labels,batch2_labels,batch3_labels,batch4_labels,batch5_labels),axis=0)

#one-hot encodinf of labels
train_labels=np.zeros((50000,10),dtype=np.float32)
test_labels=np.zeros((10000,10),dtype=np.float32)

for i in range(50000):
a=train_labels_data[i]
train_labels[i,a]=1.

for j in range(10000):
b=test_labels_data[j]
test_labels[j,b]=1.

#Défining network
def weight_variable(shape):
initial = tf.random_normal(shape, stddev=0.05)
return tf.Variable(initial)
def bias_variable(shape):
initial=tf.random_normal(0.05,shape=shape)
return tf.Variable(initial)

def conv2dstride2(x,W):
return tf.nn.conv2d(x,W,strides=[1, 2, 2, 1], padding='VALID')

def conv2d(x,W):
return tf.nn.conv2d(x,W,strides=[1, 1, 1, 1], padding='VALID')
def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

x=tf.placeholder(tf.float32, [None, None, None, 3],name="x-input")
y_=tf.placeholder(tf.float32, [None, 10],name="y-input")

keep_prob_input=tf.placeholder(tf.float32)
keep_prob_pool=tf.placeholder(tf.float32)

#First set of conv followed by conv stride 2 operation and dropout 0.5
W_conv1=weight_variable([3,3,3,96])
b_conv1=bias_variable([96])

x_dropped=tf.nn.dropout(x,keep_prob_input)
h_conv1=tf.nn.relu(conv2d(x_dropped,W_conv1)+b_conv1)

W_conv2=weight_variable([3,3,96,96])
b_conv2=bias_variable([96])

h_conv2=tf.nn.relu(conv2d(h_conv1,W_conv2)+b_conv2)

W_conv3=weight_variable([3,3,96,96])
b_conv3=bias_variable([96])

h_conv3=tf.nn.relu(conv2dstride2(h_conv2,W_conv3)+b_conv3)

h_conv3_dropped=tf.nn.dropout(h_conv3,keep_prob_pool)

#Second set of conv followed by conv stride 2 operation

W_conv4=weight_variable([3,3,96,192])
b_conv4=bias_variable([192])

h_conv4=tf.nn.relu(conv2d(h_conv3_dropped,W_conv4)+b_conv4)

W_conv5=weight_variable([3,3,192,192])
b_conv5=bias_variable([192])

h_conv5=tf.nn.relu(conv2d(h_conv4,W_conv5)+b_conv5)

W_conv7=weight_variable([3,3,192,192])
b_conv7=bias_variable([192])

h_conv7=tf.nn.relu(conv2dstride2(h_conv5,W_conv7)+b_conv7)

h_conv7_dropped=tf.nn.dropout(h_conv7,keep_prob_pool)

#Third set of conv followed by conv stride 2 operation

W_conv8=weight_variable([3,3,192,192])
b_conv8=bias_variable([192])

h_conv8=tf.nn.relu(conv2d(h_conv7_dropped,W_conv8)+b_conv8)

W_conv9=weight_variable([1,1,192,192])
b_conv9=bias_variable([192])

h_conv9=tf.nn.relu(conv2d(h_conv8,W_conv9)+b_conv9)

W_conv10=weight_variable([1,1,192,10])
b_conv10=bias_variable([10])

h_conv10=tf.nn.relu(conv2d(h_conv9,W_conv10)+b_conv10)

h_conv10_pooled=tf.nn.avg_pool(h_conv10,ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

y_conv10_reshaped=tf.reshape(h_conv10_pooled,(-1,10))

#y_conv=y_conv4_reshaped
y_conv=tf.nn.softmax(y_conv10_reshaped)

#Creating a decay for the learning rate at step 200, 250 and 300 (there is probably a better way of doing that)
NUM_EPOCHS_PER_DECAY_1=200
NUM_EPOCHS_PER_DECAY_2=250
NUM_EPOCHS_PER_DECAY_3=300
LEARNING_RATE_DECAY_FACTOR=0.1
num_batches_per_epoch=50000/128
decay_steps_1=int(num_batches_per_epoch*NUM_EPOCHS_PER_DECAY_1)
decay_steps_2=int(num_batches_per_epoch*NUM_EPOCHS_PER_DECAY_2)
decay_steps_3=int(num_batches_per_epoch*NUM_EPOCHS_PER_DECAY_3)
starter_learning_rate=0.05

global_step=tf.Variable(0, trainable=False)

decayed_learning_rate_1=tf.train.exponential_decay(starter_learning_rate,
global_step,
decay_steps_1,
LEARNING_RATE_DECAY_FACTOR,
staircase=True)
decayed_learning_rate_2=tf.train.exponential_decay(decayed_learning_rate_1,
global_step,
decay_steps_2,
LEARNING_RATE_DECAY_FACTOR,
staircase=True)
decayed_learning_rate=tf.train.exponential_decay(decayed_learning_rate_2,
global_step,
decay_steps_3,
LEARNING_RATE_DECAY_FACTOR,
staircase=True)

saver=tf.train.Saver()

sess=tf.Session()

with tf.name_scope("xent") as scope:

cross_entropy= -tf.reduce_mean(y_*tf.log(tf.clip_by_value(y_conv,10e-30,1.)))+\
(0.001/10.)*(tf.nn.l2_loss(W_conv9)+tf.nn.l2_loss(W_conv10)+\
tf.nn.l2_loss(W_conv8)+tf.nn.l2_loss(W_conv7)+\
tf.nn.l2_loss(W_conv5)+\
tf.nn.l2_loss(W_conv3)+tf.nn.l2_loss(W_conv4)+\
tf.nn.l2_loss(W_conv2)+tf.nn.l2_loss(W_conv1)+tf.nn.l2_loss(biases))
#the second term is the weight decay
ce_summ=tf.scalar_summary("cross_entropy", cross_entropy)

#We are using stochastic gradient descent with fixed momentum of 0.9
with tf.name_scope("train") as scope:
train_step=tf.train.MomentumOptimizer(decayed_learning_rate,0.9).minimize(cross_entropy,global_step)

with tf.name_scope("test") as scope:
correct_prediction =tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))

accuracy=tf.reduce_mean(tf.cast(correct_prediction,tf.float32),0)
accuracy_summary=tf.scalar_summary("accuracy",accuracy)

with tf.name_scope("Learning_rate") as scope:

tf.scalar_summary('learning_rate',decayed_learning_rate)

merged=tf.merge_all_summaries()
writer=tf.train.SummaryWriter("/cifar_logs",sess.graph_def)

#Actually training the network
init=tf.initialize_all_variables()
sess.run(init)

for i in range(150000):
batch=CIFAR.train.next_batch(128)
global_step=i
if i % 1000 == 0:
feed={x: batch[0], y_: batch[1], keep_prob_input: 1, keep_prob_pool: 1}
result=sess.run([merged,accuracy],feed_dict=feed)
summary_str=result[0]
acc=result[1]
print ("Accuracy at step %s is :%s" % (i,acc))
else:
sess.run(train_step, feed_dict={x: batch[0], y_: batch[1], keep_prob_input: 0.8, keep_prob_pool: 0.5})

save_path= saver.save(sess,"/home/model.ckpt")
print "Model saved in file: ", save_path

print "Final Accuracy on train set: "+str(sess.run(accuracy, feed_dict={x: CIFAR.train.images, y_: CIFAR.train.labels, keep_prob_input: 1, keep_prob_pool: 1}))
print "Final Accuracy on test set: "+str(sess.run(accuracy, feed_dict={x: CIFAR.test.images[range(0,500)], y_: CIFAR.test.labels[range(0,500)], keep_prob_input: 1, keep_prob_pool: 1}))

tf.Session.close(sess)


Do you see something that troubles you in my code ? I am disatisfied for exemple of my writing the weight decay in this ugly fashion. I used mean instead of sum for cross-entropy to avoid exploding ReLU grads. If you have some ideas to make the accuracy jump I am all ears.

I believe that your last pooling layer does not match the All-CNN article's architecture. They are doing a 6x6 avg pooling, it seems that your h_conv10_pooled is doing a 2x2 non-overlapping avg pooling.

A far less important point, but it is worth pointing out nonetheless: it seems that you have initialized all your weight layers using a Gaussian with standard deviation .05. I took a quick look at the All ConvNet article and they do not seem to list their weight initialization scheme. I also did not find any reference to "Glorot" or "He" (also referred to as "MSRA" or "MSR" for Microsoft Research Asia or Microsoft Research, respectively) which are very commonly used initialization schemes.

Unless I'm overlooking some small coding error, it may be that you are using a different initialization scheme than the original authors?

If fixing the pooling kernel doesn't work, you could email the authors and seeing if they could share the weight initialization scheme or try the aforementioned weight initialization schemes: Xavier Glorot and Yoshua Bengio's scheme in addition the He/MSR initialization that more recently came out; I believe both are readily available in TensorFlow. You may see some improved accuracy and some performance gains. I haven't tried benchmarking the All-CNN so I cannot be certain.

But if I had to bet the 6x6 avg pooling kernel may be the source of the inability to reproduce their experimental results.

• Thanks for your answer !!! I agree with you on initialization it is not the main problem but it is true that the authors are surprisingly silent about it. I already noticed that the pooling was not the same as the authors but the width and height of my output just before Avg-pooling was already super low and did not even have 6x6 size (I think it was 2x2 hence the size (I need to redo the calculations)). Do you think that this may be the source of the problem because I used VALID convolutions and not SAME ?. ! – jean Feb 26 '16 at 0:10
• And for the Avg pooling you are saying they do it with stride 1 and overlapping kernel ? I will look into that and come back as soon as I have some results! Thanks again. – jean Feb 26 '16 at 0:10
• @jean I am not sure to be honest, I gave the paper a cursory glance, it would be a good thing to check that all the intermediate activations in your network match what you think they should be. From what I could gather, the main architectural difference is the last pooling layer. Good luck, I hope you can fix it. Also, I wouldn't hesitate to contact the authors of the paper for some clarification. – Indie AI Feb 26 '16 at 13:44
• Thanks @Indie AI you gave only a quick glance at the paper but your comments are accurate and to the point. I must say that I was not so willing to contact the authors as, like you just showed, the error of my implementation may be obvious. But I will investigate what you pointed out. See where I get and make a decision to contact them or not afterwards. – jean Feb 26 '16 at 13:53
• In fact it looks like they use SAME convolutions and that they must get 8x8 output and not 6x6 – jean Feb 28 '16 at 13:43

I reimplemented all of it, now the accuracy on CIFAR-10 test set is at 89.31%.

The key points were :

-the preprocessing: GCN followed by ZCA-whitening in that order

-the mode of the convolutions: most of them are SAME convolutions (padding=1 for kernels of 3x3) except for the last two which are VALID, which leads to an 8x8xd output, which is then averaged.

-the initialization: small gaussian or He

-the scheduled decrease of learning rate: after 200, 250 and 300 epochs

-the size of the output the logits and softmax are of size 16 even if there are only 10 classes I guess it comes from the fact that with more logits softmax is more spread.

Nervana neon claims to have attained 89.5% on caffe and nervana but even if I would like it to match exactly this number I consider it to be close enough for my needs. Link towards my github.

I've trained an all-cnn-c model. On CIFAR-10, the test accuracy is 91.97% (there was a checkpoint with over 92%), and test loss is 0.4654. The architecture is almost the same as described in the paper, except that they didn't mention whether or not batch normalization was applied.

FYI:

Batch Normalization: After the ReLUs of all convolutional layers.

DropOut:

1. drop_rate=0.2 after input layer
2. drop_rate=0.5 after every convolutional layer with strides=2

Batch Size: 128

Initial Learning Rate: 0.01

Learning Rate Decay Scale: 0.1, applied after [200, 250, 300] epochs.

Initialization for convolutional layers: He Initialization.

Weight Decay: L2-Regularization with scale=0.001

Optimizer: momentum optimizer with momentum=0.9, use_nesterov=True

Data Augmentation: generate augmented data randomly using keras.preprocessing.image.ImageDataGenerator(zoom_range=[0.8,1.2], rotation_range=15, width_shift_range=.17, height_shift_range=.17, horizontal_flip=True)

Cross-Validation: Not in use.

• The question was about reproducing the results of the paper. – jean Jun 8 '18 at 6:44
• Sure. According to the paper one can achieve an accuracy over 90% without data augmentation and around 92.7% with "small" data augmentation. I've only used tf.layers.conv2d to construct the graph instead of building everything (such as get the weight and bias variables) from scratch and hence i'm not familiar with your approach. But my point is you can't go wrong by 1% if your architecture is identical to the one the article uses. For me the zca whitening does not contribute much to the overall accuracy. – Shihao Xu Jun 8 '18 at 16:25