I've been trying to code a neural network library in Java which contains very simple stuff like a Convolution-layer, Dense-Layer and a Flattening-Layer.
I am pretty sure that my Dense-layer works as expected but I am not sure about the convolution. My network architecture looks like this:
NetworkBuilder builder = new NetworkBuilder(1, 28, 28); builder.addLayer(new ConvLayer(8, 3, 1, 2) //act.-maps, filter_size, stride, padding .biasRange(0, 0) .weightsRange(-2, 2) .setActivationFunction(new ReLU())); builder.addLayer(new PoolingLayer(2)); //downsampling factor builder.addLayer(new ConvLayer(26, 5, 1, 0) //act.-maps, filter_size, stride, padding .biasRange(0, 0) .weightsRange(-2, 2) .setActivationFunction(new TanH())); builder.addLayer(new PoolingLayer(2)); //downsampling factor builder.addLayer(new TransformationLayer()); //Flattenig (3D -> 1D) builder.addLayer(new DenseLayer(120) //120 neurons .setActivationFunction(new Sigmoid()) ); builder.addLayer(new DenseLayer(10) .setActivationFunction(new Softmax()) ); Network network = builder.buildNetwork(); network.setErrorFunction(new CrossEntropy());
- 500 images to train on.
N iterations with 5 images each time selected randomly and trained with a learning rate of 0.3 * overall_error.
Stochastic gradient descent with the backpropagation algorithm
My network can learn up to 500 images perfectly but when I go higher than that, my accuracy goes down. (Probably not enough activation maps or sth.like that)
When I take more input data, it might happen that my overall error reduces first, but ends up increasing towards infinity
The filter of my first convolution layer look like this:
0,26756 0,3341 0,57849 -0,40428 0,10674 0,53553 (filter 1) -0,27761 0,17577 -0,32525 177,92366 186,30331 -173,91031 -74,38991 1,29136 28,72685 (filter 5) 72,06218 37,42708 27,14726 . . .
Is it a normal thing that some filters have values close to zero and some other filters have values up to 1 million? (I had that aswell)
If I use a ReLU activation function in the second convolution layer instead of the TanH activation function, my network error does not reduce at all and my accuracy is around 10% (on the Mnist dataset) -> pretty much random
If those things that I mentioned above are things that can sometimes happen in neural networks, I assume my implementation is correct but my training parameters are wrong but if not, I need to check my code again.