Unsure if my implementation of a Convolutional layer doesn't learn or it's the correct behaviour

Question

So for the last week or so I've tried to implement a Feed Forward network with multiple types of layers (Fully Connected, MaxPool, Convolution), multiple types of non-linear functions (tanh, sigmoid, relu), using the gradient descent optimizer and 2 errors types (sum of squared errors and logistic error). It should be noted that I'm really new in the domain, so I might be making some obvious mistakes I'm not even aware of, which is one of the things I'm looking to fix with this post.

The implementation can be found here, which is in Python 3, completely unoptmized (not the purpose of the project) and the main function is in main.py.

The training is done only on the MNIST dataset (though my plans were to use it for CIFAR-10 as well, but in the current form it's quite obvious I can't do it). The images in the MNIST dataset are 1x32x32 (grayscale and a padding of 2 from their original shape).

The pictures I'll be putting are comparing the accuracy and computed error based on an evaluation set that is randomly picked from the training data (while training, random is redone every time, check evaluate function in hw2.py). The error calculation is done on the respective formulas for both kinds of errors, and can be found on error.py file. The weights/biases are initialized using Xavier method (2 / volume size) for both Fully Connected and Convolutional layers (layer.py).

All the tests I show below are using the Sum of Squared errors.

Regular 2-layered network

The architecture is as follows: 0 Linearize (1, 32, 32) -> (1024, 1) 1 FullyConnected (1024, 1) -> (100, 1) (sigmoid) 2 FullyConnected (100, 1) -> (10, 1) The learning rate used: 0.001 (1e-3).

This network gives me a pretty good result (inexplicably good, as described even by LeCun, page 3 at the end).

Here are the results for the simple 2 layers network, which look really well compared to what I'll show afterwards:

The testing phase with this trained network gives me 85-90% accuracy results, which I consider really good (Tensorflow considers this embarrassing in their tutorial :) ).

Convolutional network

So, as many tutorials/documentation that I read says, adding a Convolutional layer should improve the accuracy of my testing as the layer learns filters specifically for the dataset instead of having 1 weight mapping between each neuron in the 2 layers. Anyway, I've implemented (with a lot of headache) a Convolutional layer (2 implementations: one simply a 6-for translation of the derivatives formula for forward/backpropagation and one slightly vectorized, still very slow, again not the purpose of the project). The implementation is in layer.py.

For this implementation I used some unit testing on the inputs and outputs of the forward and backward step (to validate my 6-for implementation with the vectorized one as well as the results themselves). The test called testConvolution is identical to the convolution test done in the implementation of tiny-dnn (a C++ header-only neural network framework which can be found here), more specifically TEST(convolutional, with_stride), which seems the only test that does not check their correctness between vectorized and linear implementation, but the correctness of the outputs w.r.t inputs.

For the second network, I used the following architecture: 0 Convolution (1, 32, 32) -> (5, 28, 28) (tanh) 1 Linearize (5, 28, 28) -> (3920, 1) 2 FullyConnected (3920, 1) -> (100, 1) (sigmoid) 3 FullyConnected (100, 1) -> (10, 1)

So the convolutional layer has 5 filters of shape 5x5, with a stride of 1. The learning rate I used was 0.0005 (5 * 1e-4). The training was done in 2 epochs (values on X axis after 60000 are from second epoch). The training took about 14 hours. The results are as follows:

So, as we can interpret from the plotted results, the error kind of stabilized around the value of 50, while the accuracy is all over the place (sometimes giving me 0.0%, sometimes going to 70%, peaking at 90% with 2 runner ups of 85%). Now these are results I cannot really understand entirely. The network "seems" to be learning something, but in the same time it gives seemingly random results, such as going from 0.1% to 0.7% between two (random) evaluations very often. The error itself stops decreasing after a point and never goes below a certain value, compared to the simple network, where the error always goes downward due to gradient descent.

The testing phase of the final model with this network yielded an accuracy of 51%. I'm sure the Tensorflow guys would be stoked.

Convolutional network - LeNet-5

Alright, so I trained this network in parallel with the one above, and while the above one finished (14 hours as I said), this one is still going, but I took the preliminary results I have so far which seem concludent of what's going to happen at the end of the training phase.

My theory was that perhaps since the network above was giving me some decent results (sometimes evaluating above 70%, while sometimes 0-10%).

The network isn't 100% LeNet-5 (I don't make the partial convolution they do in the paper [lecun-98.pdf]): 0 Convolution (1, 32, 32) -> (6, 28, 28) (tanh) 1 MaxPool (6, 28, 28) -> (6, 14, 14) 2 Convolution (6, 14, 14) -> (16, 10, 10) (tanh) 3 MaxPool (16, 10, 10) -> (16, 5, 5) 4 Convolution (16, 5, 5) -> (120, 1, 1) (tanh) 5 Linearize (120, 1, 1) -> (120, 1) 6 FullyConnected (120, 1) -> (86, 1) (tanh or sigmoid, can't remember) 7 FullyConnected (86, 1) -> (10, 1)

The learning rate is also 0.0005 (5 * 1e-4). The results are as follows (everything after 60000 on x axis is also from 2nd epoch, which hasn't finished yet):

As we can see, the error stabilized around the value of 52-53, while the accuracy itself is all over the place (similar to previous network), but never going above 30-40%.

My actual questions

Ok, so after writing all the stuff, I guess I am ready to ask you if you can clarify some things for me based on these results.

1. Are the results I get correct? I'm saying this because tiny-dnn has an example of a network training on MNIST and after 1 epoch their result is ~70-80% using the same network (LeNet-5) and the same optimizer (gradient descent), while for my it's stuck at 20-30% with the error not changing at all.
2. What could be the reason that the regular network (no convolution) outperforms the convolutional implementation so much ? Could it be a bad implementation on my side and if so, is there any reference implementation I could look at (I'm trying to reverse-understand the tiny-dnn implementation and this seems to be my next step)
3. Perhaps you may be wondering why I'm using such "precise" learning rates. Well, had I used even slightly bigger ones (1e-3 for convolution), after 30-40 images, my error would look something like this (followed by a nasty error):
4. Should I normalize my weights? If yes, what do I do? The only place I update my weights/biases are in the backpropagation step, after a batch of images, calling the optimizer (weights -= learningRate * gradientWeights), where gradientWeights is the accumulation of gradients of a batch (layer.py and optimizer.py). I'm afraid updating the weights in another way might influence the gradient descent algorithm and give me unreliable results. Also, the tiny-dnn (my first impression only), only update the weights (not the biases), but I might be wrong as I didn't dig very deep in their code (optimizers/optimizer.h).
5. Should I add a regularization term to my errors (see error.py) and if yes, how would I go about that? Do I just update the derivation of the error at the last layer (derive the sum of the weights w.r.t weights and biases as well). What formula do I use for regularization, do I sum all the weights in the network (so the weights of layers 0, 2, 4, 6, 7 in LeNet-5, as only those have trainable parameters) or just the last layer?
6. What should be my general next steps?

If anyone's reading this, thanks.

Answer 1

Yes, indeed it was a couple of coding errors rather than a network model problem. More specifically here, I have a "Linearize layer" which reshapes my tensors from one shape to another (convolution from 3D shapes to column shapes for fully connected) where I was backpropagating the output, rather than the error :)

Anyway, here are the results for MNIST and CIFAR-10 for the following model: 0 Convolution (1 [for MNIST] / 3 for CIFAR-10], 32, 32) -> (6, 28, 28) (Tanh) 1 MaxPool (6, 28, 28) -> (6, 14, 14) 2 Convolution (6, 14, 14) -> (16, 10, 10) (Tanh) 3 MaxPool (16, 10, 10) -> (16, 5, 5) 4 Convolution (16, 5, 5) -> (120, 1, 1) (Tanh) 5 Linearize (120, 1, 1) -> (120, 1) 6 FullyConnected (120, 1) -> (10, 1) (Tanh) 7 SoftMax (10, 1) -> (10, 1)

Learning rate is 0.001 and there are 2 epochs (after 60k in mnist and after 50k in cifar-10 2nd epoch starts)

MNIST

Test accuracy: 98%

CIFAR-10

Test accuracy: 42-45%.

My next steps are to implement the algorithms recommended above by Lukasz and to check if the error plot is actually correct as it seems a bit weird. I'll also try to vectorize the code much better (I'm thinking a straight forward update would be to process a batch of images at once and forward/backward a vector of results for each image).