CIFAR-10 Can't get above 60% Accuracy, Keras with Tensorflow backend [closed]

Training after 15 epochs on the CIFAR-10 dataset seems to make the validation loss no longer decrease, sticking around 1.4 (with 60% validation accuracy). I've shuffled the training set, divided it by 255, and imported as float32. I've tried numerous architectures, both with and without dropout in the Conv2D layers and nothing seems to work. The same architecture achieves 99.7% accuracy on test sets for MNIST. Please see the architecture below:

(Note: I have tried increasing dropout and increasing/decreasing learning rate of the Adam optimizer to prevent overfitting, all this does is prevent overfitting but with both training and test set now having similar low accuracy around 60%).

with tf.device('/gpu:0'):
tf.placeholder(tf.float32, shape=(None, 20, 64))
#placeholder initialized (pick /cpu:0 or /gpu:0)
seed = 6
np.random.seed(seed)
modelnn = Sequential()
neurons = x_train_reduced.shape[1:]

modelnn.add(Convolution2D(32, 3, 3, input_shape=neurons, activation='relu', border_mode='same'))
modelnn.compile(loss='categorical_crossentropy', optimizer=optimizer_input, metrics=['accuracy'])
y_train = to_categorical(y_train)
modelnn.fit(x_train_reduced, y_train, nb_epoch=nb_epoch_count, shuffle=True, batch_size=bsize,
validation_split=0.1)


Results:

    44100/44100 [==============================] - 22s - loss: 2.1453 - acc: 0.2010 - val_loss: 1.9812 - val_acc: 0.2959
Epoch 2/50
44100/44100 [==============================] - 24s - loss: 1.9486 - acc: 0.3089 - val_loss: 1.8685 - val_acc: 0.3567
Epoch 3/50
44100/44100 [==============================] - 18s - loss: 1.8599 - acc: 0.3575 - val_loss: 1.7822 - val_acc: 0.3982
Epoch 4/50
44100/44100 [==============================] - 18s - loss: 1.7925 - acc: 0.3933 - val_loss: 1.7272 - val_acc: 0.4229
Epoch 5/50
44100/44100 [==============================] - 18s - loss: 1.7425 - acc: 0.4195 - val_loss: 1.6806 - val_acc: 0.4459
Epoch 6/50
44100/44100 [==============================] - 18s - loss: 1.6998 - acc: 0.4440 - val_loss: 1.6436 - val_acc: 0.4682
Epoch 7/50
44100/44100 [==============================] - 18s - loss: 1.6636 - acc: 0.4603 - val_loss: 1.6156 - val_acc: 0.4837
Epoch 8/50
44100/44100 [==============================] - 18s - loss: 1.6333 - acc: 0.4781 - val_loss: 1.6351 - val_acc: 0.4776
Epoch 9/50
44100/44100 [==============================] - 18s - loss: 1.6086 - acc: 0.4898 - val_loss: 1.5732 - val_acc: 0.5063
Epoch 10/50
44100/44100 [==============================] - 18s - loss: 1.5776 - acc: 0.5065 - val_loss: 1.5411 - val_acc: 0.5227
Epoch 11/50
44100/44100 [==============================] - 18s - loss: 1.5585 - acc: 0.5145 - val_loss: 1.5485 - val_acc: 0.5212
Epoch 12/50
44100/44100 [==============================] - 18s - loss: 1.5321 - acc: 0.5288 - val_loss: 1.5354 - val_acc: 0.5316
Epoch 13/50
44100/44100 [==============================] - 18s - loss: 1.5082 - acc: 0.5402 - val_loss: 1.5022 - val_acc: 0.5427
Epoch 14/50
44100/44100 [==============================] - 18s - loss: 1.4945 - acc: 0.5438 - val_loss: 1.4916 - val_acc: 0.5490
Epoch 15/50
44100/44100 [==============================] - 192s - loss: 1.4762 - acc: 0.5535 - val_loss: 1.5159 - val_acc: 0.5394
Epoch 16/50
44100/44100 [==============================] - 18s - loss: 1.4577 - acc: 0.5620 - val_loss: 1.5389 - val_acc: 0.5257
Epoch 17/50
44100/44100 [==============================] - 18s - loss: 1.4425 - acc: 0.5671 - val_loss: 1.4590 - val_acc: 0.5667
Epoch 18/50
44100/44100 [==============================] - 18s - loss: 1.4258 - acc: 0.5766 - val_loss: 1.4552 - val_acc: 0.5763
Epoch 19/50
44100/44100 [==============================] - 18s - loss: 1.4113 - acc: 0.5805 - val_loss: 1.4439 - val_acc: 0.5767
Epoch 20/50
44100/44100 [==============================] - 18s - loss: 1.3971 - acc: 0.5879 - val_loss: 1.4473 - val_acc: 0.5769
Epoch 21/50
44100/44100 [==============================] - 18s - loss: 1.3850 - acc: 0.5919 - val_loss: 1.4251 - val_acc: 0.5871
Epoch 22/50
44100/44100 [==============================] - 18s - loss: 1.3668 - acc: 0.6006 - val_loss: 1.4203 - val_acc: 0.5910
Epoch 23/50
44100/44100 [==============================] - 18s - loss: 1.3549 - acc: 0.6051 - val_loss: 1.4207 - val_acc: 0.5939
Epoch 24/50
44100/44100 [==============================] - 18s - loss: 1.3373 - acc: 0.6111 - val_loss: 1.4516 - val_acc: 0.5784
Epoch 25/50
44100/44100 [==============================] - 18s - loss: 1.3285 - acc: 0.6149 - val_loss: 1.4146 - val_acc: 0.5922
Epoch 26/50
44100/44100 [==============================] - 18s - loss: 1.3134 - acc: 0.6205 - val_loss: 1.4090 - val_acc: 0.6024
Epoch 27/50
44100/44100 [==============================] - 18s - loss: 1.3043 - acc: 0.6239 - val_loss: 1.4307 - val_acc: 0.5959
Epoch 28/50
44100/44100 [==============================] - 18s - loss: 1.2862 - acc: 0.6297 - val_loss: 1.4241 - val_acc: 0.5978
Epoch 29/50
44100/44100 [==============================] - 18s - loss: 1.2706 - acc: 0.6340 - val_loss: 1.4046 - val_acc: 0.6067
Epoch 30/50
44100/44100 [==============================] - 18s - loss: 1.2634 - acc: 0.6405 - val_loss: 1.4120 - val_acc: 0.6037
Epoch 31/50
44100/44100 [==============================] - 18s - loss: 1.2473 - acc: 0.6446 - val_loss: 1.4067 - val_acc: 0.6045
Epoch 32/50
44100/44100 [==============================] - 18s - loss: 1.2411 - acc: 0.6471 - val_loss: 1.4083 - val_acc: 0.6098
Epoch 33/50
44100/44100 [==============================] - 18s - loss: 1.2241 - acc: 0.6498 - val_loss: 1.4091 - val_acc: 0.6076
Epoch 34/50
44100/44100 [==============================] - 18s - loss: 1.2121 - acc: 0.6541 - val_loss: 1.4209 - val_acc: 0.6127
Epoch 35/50
44100/44100 [==============================] - 18s - loss: 1.1995 - acc: 0.6582 - val_loss: 1.4230 - val_acc: 0.6131
Epoch 36/50
44100/44100 [==============================] - 18s - loss: 1.1884 - acc: 0.6622 - val_loss: 1.4024 - val_acc: 0.6124
Epoch 37/50
44100/44100 [==============================] - 18s - loss: 1.1778 - acc: 0.6657 - val_loss: 1.4328 - val_acc: 0.6080
Epoch 38/50
44100/44100 [==============================] - 18s - loss: 1.1612 - acc: 0.6683 - val_loss: 1.4246 - val_acc: 0.6159
Epoch 39/50
44100/44100 [==============================] - 18s - loss: 1.1466 - acc: 0.6735 - val_loss: 1.4282 - val_acc: 0.6122
Epoch 40/50
44100/44100 [==============================] - 18s - loss: 1.1325 - acc: 0.6783 - val_loss: 1.4311 - val_acc: 0.6157
Epoch 41/50
44100/44100 [==============================] - 18s - loss: 1.1213 - acc: 0.6806 - val_loss: 1.4647 - val_acc: 0.6047
Epoch 42/50
44100/44100 [==============================] - 18s - loss: 1.1064 - acc: 0.6842 - val_loss: 1.4631 - val_acc: 0.6047
Epoch 43/50
44100/44100 [==============================] - 18s - loss: 1.0967 - acc: 0.6870 - val_loss: 1.4535 - val_acc: 0.6106
Epoch 44/50
44100/44100 [==============================] - 18s - loss: 1.0822 - acc: 0.6893 - val_loss: 1.4532 - val_acc: 0.6149
Epoch 45/50
44100/44100 [==============================] - 18s - loss: 1.0659 - acc: 0.6941 - val_loss: 1.4691 - val_acc: 0.6108
Epoch 46/50
44100/44100 [==============================] - 18s - loss: 1.0610 - acc: 0.6956 - val_loss: 1.4751 - val_acc: 0.6106
Epoch 47/50
44100/44100 [==============================] - 18s - loss: 1.0397 - acc: 0.6981 - val_loss: 1.4857 - val_acc: 0.6041
Epoch 48/50
44100/44100 [==============================] - 18s - loss: 1.0208 - acc: 0.7039 - val_loss: 1.4901 - val_acc: 0.6106
Epoch 49/50
44100/44100 [==============================] - 18s - loss: 1.0187 - acc: 0.7036 - val_loss: 1.4994 - val_acc: 0.6106
Epoch 50/50
44100/44100 [==============================] - 18s - loss: 1.0024 - acc: 0.7070 - val_loss: 1.5078 - val_acc: 0.6039
Time: 1109.7512991428375
Neural Network now trained from dimensions (49000, 3, 32, 32)


Update: Further testing including BatchNormalization both with and without MaxNorm -

New architecture:

    modelnn.add(Convolution2D(32, 3, 3, input_shape=neurons, activation='relu', border_mode='same'))
# modelnn.add(Convolution2D(256, 3, 3, activation='relu', border_mode='same'))
# modelnn.add(Convolution2D(256, 3, 3, activation='relu', border_mode='same'))


closed as off-topic by mkt, kjetil b halvorsen, mdewey, whuber♦Feb 7 at 22:31

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – mkt, kjetil b halvorsen, mdewey, whuber
If this question can be reworded to fit the rules in the help center, please edit the question.

Note that MNIST is a much simpler problem set than CIFAR-10, and you can get 98% from a fully-connected (non-convolutional) NNet with very little difficulty. A very simple CNN with just one or two convolutional layers can likewise get to the same level of accuracy.

I'm not sure about your NNet architecture, but I can get you to 78% test accuracy on CIFAR-10 with the following architecture (which is comparatively simpler and has fewer weights). No special initialization or handholding was required, using vanilla defaults and Adam optimizer:

model = Sequential()


This architecture is pretty simple, and is loosely based on https://arxiv.org/pdf/1412.6806.pdf.

Training this model thusly:

n_epochs = 25
batch_size = 256
callbacks_list = None
H = model.fit(trainX, trainY, validation_data=(testX, testY),
epochs=n_epochs, batch_size=batch_size, callbacks=callbacks_list)
print('Done!!!')


Gives the following, which you can see gets to almost 77% by the 25th epoch and more or less flattens out from there (but has enough regularization from dropout to prevent it from degrading due to overfitting, at least over the tested number of iterations).

Train on 50000 samples, validate on 10000 samples
Epoch 1/50
50000/50000 [==============================] - 19s 390us/step - loss: 1.6058 - acc: 0.4150 - val_loss: 1.5285 - val_acc: 0.4669
Epoch 2/50
50000/50000 [==============================] - 19s 371us/step - loss: 1.2563 - acc: 0.5477 - val_loss: 1.1447 - val_acc: 0.5901
Epoch 3/50
50000/50000 [==============================] - 19s 373us/step - loss: 1.0784 - acc: 0.6163 - val_loss: 1.1577 - val_acc: 0.6002
...
Epoch 25/50
50000/50000 [==============================] - 19s 374us/step - loss: 0.3188 - acc: 0.8857 - val_loss: 0.7493 - val_acc: 0.7680
...
Epoch 50/50
50000/50000 [==============================] - 19s 373us/step - loss: 0.1928 - acc: 0.9329 - val_loss: 0.8718 - val_acc: 0.7751
Done!!!

Here's an even simpler and much smaller architecture that can get to 70% pretty quickly with the same training regimen (no BatchNormalization or pooling layers):

# CNN architecture with Keras
model = Sequential()
use_bias=True, kernel_size=(3,3)))


It's worth noting that the architectures that get to best-published accuracy on CIFAR-10 (currently in the 90-96% range) are generally more complicated and take many hours to train on GPU hardware. But I've been able to get to the 70-80% range with fairly simple architectures that train in minutes, which is what I'd recommend prior to going for best-published results which usually require more complicated architectures, longer training periods, sometimes special handholding/training regimens or data augmentation, and hours of train time.

UPDATE:

Based on the updated plots in the question, the most obvious problem that is demonstrated is overfitting. This is evidenced by the divergence of the train-test data after about the 15th epoch which demonstrates insufficient regularization for this architecture, for this data set. You will be unlikely to get improvement from tuning any other hyperparameters (normalization strategies, learning rates, etc.) unless the overfitting is addressed.

In using NNets, I recommend the following:

1. Start from architectures that mimic or replicate those known to produce good results
2. Verify performance on your data set, with a particular eye to over-fitting in the network (evidenced by significant divergence of train-test errors)
3. Add additional regularization (increase dropout rates) when overfitting is observed (you're looking for "just enough" to prevent overfitting - too much will result in under-fitting)
4. Experiment with structure, training approaches, and hyper-parameters to find avenues of improvement

Prescriptions regarding the latter are actually quite hard to come by, because there is little theoretical foundation for how structure, training, or hyper-parameters interact to yield performance on any given data set. That the approaches employed by published architectures achieving similarly-high levels of performance on benchmark data sets vary so greatly is evidence of this.

Batchnormalization has been found to significantly improve some architectures, but others can do quite well without it (or are indifferent to its presence). The only real guidance to provide here is to try it and see if it helps.

Fine-tuning learning rates should generally be avoided unless you are an advanced practitioner with a deep understanding of ConvNets and the corresponding ability to read the tea leaves with regard to inter-epoch incremental performance during training. Customized learning rates and other specialized training regimens can in some cases can help networks navigate around local minima and find better overall solutions, but unless you have a lot of time and the know-how to diagnose the convergence behavior of the network, this is not a good place to start. Most of us should use an optimizer like Adam which will outperform novice attempts at hand-tuned learning rates in the vast majority of cases.

Data augmentation via image preprocessing can sometimes yield significant performance improvements (in general, the more varied the input data the better the model will generalize - data preprocessing adds variation to the input space which can improve out-of-sample accuracy and may afford reduction in regularization requirements - hypothetically, with infinite training data we wouldn't need any regularization at all, but in the image processing space we are unlikely to approach that asymptote). This can significantly increase training time and slow convergence rates though, and introduces a whole other set of hyperparameters relating to input image permutation techniques (rotation, cropping, scaling, noising, etc. etc.). Because this path can increase training times and require additional experiments to tune results, some general advice would be to drive for best accuracy in your network without augmentation first, then see if some modest augmentation yields improvement. If it does, it may warrant further experimentation.

For any and all tuning experiments, you will need to keep an eye out for changes in over- and under-fitting behavior. Changing network architecture, training regimens, or hyperparameters may require additional tuning of dropout regularization. The ability to readily ascertain over- and under-fitting behavior from train/test performance is arguably the most important baseline skill in working with NNets, and this becomes more intuitive with experience.

This is the candle by which all your efforts will be guided. The candle can only dimly illuminate the path, but without it you'll be stumbling around in the dark. If your network is badly over- or under-fitting, that should be addressed before attempting random permutations of network structure or hyperparameters.

The comparatively simple architectures with vanilla training regimens included in this answer demonstrate a reality of working with NNET architectures on hard problems like image classification: attaining a "pretty good" result based on approaches that are known to work well is not difficult, but incremental improvement is increasingly costly. Achieving best-published results via experimentation is going to be beyond the abilities or time availability of many (although it is possible, with enough time and effort, to follow the cookbook recipes of published approaches to replicate their results - but even this is by no means trivial). Attaining incremental improvement from a "pretty good" starting point can be a very time-consuming process of trial-and-error, and many experiments will not yield any significant improvement.

This is not meant to dissuade anyone from attempting to learn, but only to make clear that there is a significant investment required to master the (ever expanding) toolset in the NNet bag of tricks, and driving improvements through trial-and-error can require dozens (or hundreds) of experiments over days or weeks of dedicated-GPU training.

The time, skill, and resources (dedicated GPU) required to train networks to very high levels of performance explain in part the popularity of pre-trained networks.

• You neural network architectures do not have any pooling layers? doesn't that create unmanageable number of parameters ? – Spandy May 11 '18 at 7:58
• Pooling - pfft! Overrated. This uses an all-convolutional approach that uses striding for decimation rather than pooling - see the linked document for description. Convolutional striding can get the same "funneling" effect as pooling by slightly different means. That they both work simply illustrates that there's not a lot of firm theoretical ground to stand on with regard to why any of this stuff works. – T3am5hark May 11 '18 at 21:33

Looking at your in-sample and out-of-sample loss and accuracy values, your model is currently underfitted, but it is monotonically improving. In other words, it seems like running this for more epochs would result in higher predictive performance / less entropy loss.

You are using a highly regularised (drop-out layers) architecture, which is not bad. However, it is also not surprising that the training takes much longer than without any regularisation. Due to the drop-out layers, it is unlikely that you will (substantially) overfit.

Things you can try to accelerate learning:

i. tweak the learning rate: e.g. start with a small one, hike it up in the middle, and towards the end lower it down again.

ii. add batchnormalisation: in the architecture above, you can include batch-norm both in your convolutional blocks and dense layers. Usually, the batch-norm layer is added after the nonlinear activation but before dropout. I am not sure how well batch-norm plays with maxnorm. For your dense layers, I would try batch-norm+dropuout with/without maxnorm. I have a feeling you do not need maxnorm if you apply batch normalisation.

iii. increase batch-size: I am not sure what your batch-size is and whether you own a GPU. If you have a GPU, you probably should try to max your batch-size in multiplicatives of 32.

Finally, to ensure that your data is 'learnable' / not corrupt (e.g. you have not unwillingly applied a transformation to warp it), I would throw away all regularisation from your architecture, run training and see that you can overfit to the training set. If you can learn training data successfully, the rest is a generalisation exercise. If you can not overfit to training data even with no regularisation, most likely your model needs more capacity (deeper and wider architecture).

• Thank you kindly for the advice! You were right about MaxNorm interfering slightly. Nevertheless, even after adding the BatchNormalization layers (please see update) and both removing/including MaxNorm, the accuracy is still low. No augmentation is taking place either. I have a GPU, and have tried training at 64,128,256 and 512 batches but little difference is noticed. Regarding the learning rate, I'm using the Adam optimizer and thought this should more or less be left alone? Nevertehless I tried LR at 0.05, 0.001, 0.0005 and noticed the default 0.001 seems best still. Any ideas? – user4779 Apr 10 '17 at 12:51
• Also I am able to overfit fine. I tried my best to copy the architecture in the papers that seem to be able to achieve a 80%++ accuracy with MNIST10. When I leave the model training for longer epochs it seems loss now increases (more than 20 epochs or so). – user4779 Apr 10 '17 at 12:53
• Modifying comment - following the changes to the model, the graphs now indicate that it is significantly overfitting the data (based on the divergence of the validation error after ~15 epochs) – T3am5hark May 11 '18 at 21:44
• I actually dispute the utility of the advice on offer here, especially for new practitioners. These are all things you can do, sure, but for folks who are new to CNNs and don't have the intuition or understanding of how these things work, it's far too many knobs and levers to tweak without any prescriptive guidance other than blind trial and error, unlikely to yield positive results. Better would be to first start with simpler architectures that are able to get good (not best-published) performance with minimal twiddling, then explore avenues of improvement from there. My two cents. – T3am5hark May 11 '18 at 21:53
• To further elaborate - don't play with learning rates, use Adam. It's going to beat hand-tuning of learning rates 99.99% of the time. Also - the statement that it's "unlikely that you will ever overfit" is just plain wrong (as indicated by the follow-up graphics which now demonstrate significant over-fitting), and there's no good reason for the poster to assume that... there's nothing to tell you a priori for a given architecture whether a given dropout rate will sufficiently regularize to prevent over-fitting. – T3am5hark May 11 '18 at 22:02

I gave this a shot today and was consistently able to hit near 75-80% in test accuracy.

• The total number of parameters used was: 183,242

• You can do better by adding maybe a few more layers, but you don't need to be excessive. More complex networks do not always result in better results.

Suggestions

My suggestion to you is that you keep your architecture simple. Follow Occam's Razor, simple is better.

• Don't use a random seed

• Use an appropriate optimizer; I used Adadelta as is from Keras.

• CNNs don't need to be convoluted; keep it simple

• Deeper skinnier networks sometimes work better than wider ones

• Use regularization (e.g. Dropout)

Below is my code (using Keras)

# Define the model
model = Sequential()