KNN outperforms CNN

Disclaimer: I am a programmer by trade, not a statistician, so please cater to my ignorance when explaining things and I apologize now if I make any incorrect assumptions

I am currently attempting to build an OCR platform for printed characters moving at speed in a video stream. I am able to detect and segment the images like so:

These are labeled using a standard [0,0,1,0,0,0,0,0,0,0] format.

I first attempted to build a convolution neural network using keras for performing the task of recognition with the following architecture:

# First convolution layer
model = Sequential()
model.add(Convolution2D(20, 15, 15, border_mode="same",input_shape=(height, width, depth)))

# Second convolution layer

# Third convolution layer

# Fully connected layer

# Classifier

opt = SGD(lr=0.01)
model.compile(loss="categorical_crossentropy", optimizer=opt,metrics=["accuracy"])
history = model.fit(trainingData, trainingLabels, batch_size=128, epochs=150,verbose=1)


However it would appear the network converges after only a few epochs with an awful accuracy level, then stays at that level indefinitely.

I have attempted tweaking the learning rate, amount of layers, size/amount of filters but still have the same results.

At first I assumed it was down to the validity of my training data, however after training a KNN classifier on the same data it achieves 94.87% accuracy.

I originally followed this fantastic tutorial for building the architecture as it solves a similar problem (MNIST dataset)

I was hoping to use a CNN as a learning exercise into why CNN's work so well for this kind of problem, any assistance in understanding why my CNN didn't work would be greatly appreciated.

• The very first question to ask is how much data you have. Neural nets are good, but they're not magic -- different dataset sizes, image resolutions and model architectures may result in completely different outcomes. Could you give some more information regarding how much data you have and what other classifiers and neural net architectures you have tried? May 25, 2017 at 11:09
• Hey Pedro, Thanks for your time. I have a relatively small dataset, currently using 1572 images for training and 510 for testing. Images are all 81 x 127 greyscale. I have only tried the convolutional network defined above and a KNN May 25, 2017 at 11:50
• Minor adjustments to the architecture above are have 2-4 layers rather than the three above, as well as size and amounts of the filters on each layer. May 25, 2017 at 11:52
• Your dataset is way too small to fit a CNN easily. To take the MNIST dataset as an example, it has 30 times more samples and the images are 13 times smaller, which is a big difference. If that answers your question, I'd say that "not enough data" is the main reason why your CNN doesn't work. I can go through a more detailed discussion and post it as an answer if you want. May 25, 2017 at 12:35
• As a separate note, I have to ask: do you actually need more accuracy? If you're building this classifier for a specific application then the 95% of the kNN might be good enough. If you're just doing this for fun that's another matter, of course. May 25, 2017 at 12:38

Almost certainly the low performance of your CNN is due to insufficient data.

A quick double-check in Keras using model.count_params() says your network has more than 10 million parameters -- which is not too much by modern standards but is quite a big bunch if you only have 1.5k images. Conventional wisdom in ML says that you should have at least a handful of thousands of images per class if you want to consider deep learning -- although in my experience I'd say it has to be quite a bit more unless you're willing to spend a long while fine tuning your model.

If you want to go the neural net way, I would suggest you to make your network smaller and add some strong regularisation, potentially through heavy dropout or L2 regularisation. If you're serious about this you can even consider doing some data augmentation or transfer learning (potentially from MNIST).

If you're just hacking around some ML for fun, I would recommend you to look into other classifiers that are more likely to work in your scenario. A couple of examples are Support Vector Machines and Random Forests.

Indeed, as Pedro suggested your network is too big for the data, but there are also problems with the data itself:

1. The fully connected layer alone is ~10 M parameters for 16 M data points, which is guaranteed to brutally overfit, I'd guess no more than 15% accuracy for this. For such small datasets, avoid large fully connected layers. Better go all-convolutional ie. conv-pool-conv-pool-... until you have fewxfewxN_channels (few < 4). Then you can have a small FC layer before the softmax.
2. The overfiting is especially true, as the pixels in your images are far from independent, your numbers are visibly pixelated. You can easily bin them 4x4 to 20x32 pixels without losing relevant information. So bin your data and switch to 3x3 or 5x5 convolutions with pooling.
3. You can have decent results even with such small data, so don't loose hope. Just minimize the number of parameters.
• Hi Imoha, thanks very much for this! I will take your comments on board and try to amend the CNN when I return to work. May 29, 2017 at 19:28