# CNN: Range of filters and activation functions

I've been trying to code a neural network library in Java which contains very simple stuff like a Convolution-layer, Dense-Layer and a Flattening-Layer.

I am pretty sure that my Dense-layer works as expected but I am not sure about the convolution. My network architecture looks like this:

    NetworkBuilder builder = new NetworkBuilder(1, 28, 28);
.biasRange(0, 0)
.weightsRange(-2, 2)
.setActivationFunction(new ReLU()));
.biasRange(0, 0)
.weightsRange(-2, 2)
.setActivationFunction(new TanH()));
builder.addLayer(new TransformationLayer()); //Flattenig (3D -> 1D)
.setActivationFunction(new Sigmoid())
);
.setActivationFunction(new Softmax())
);
Network network = builder.buildNetwork();
network.setErrorFunction(new CrossEntropy());


Training:

• 500 images to train on.
• N iterations with 5 images each time selected randomly and trained with a learning rate of 0.3 * overall_error.

• Stochastic gradient descent with the backpropagation algorithm

Problems:

1. My network can learn up to 500 images perfectly but when I go higher than that, my accuracy goes down. (Probably not enough activation maps or sth.like that)

2. When I take more input data, it might happen that my overall error reduces first, but ends up increasing towards infinity

3. The filter of my first convolution layer look like this:

0,26756  0,3341   0,57849
-0,40428 0,10674  0,53553        (filter 1)
-0,27761 0,17577  -0,32525

177,92366 186,30331 -173,91031
-74,38991 1,29136  28,72685      (filter 5)
72,06218 37,42708 27,14726

.
.
.


Is it a normal thing that some filters have values close to zero and some other filters have values up to 1 million? (I had that aswell)

4. If I use a ReLU activation function in the second convolution layer instead of the TanH activation function, my network error does not reduce at all and my accuracy is around 10% (on the Mnist dataset) -> pretty much random

If those things that I mentioned above are things that can sometimes happen in neural networks, I assume my implementation is correct but my training parameters are wrong but if not, I need to check my code again.

• All seem caused by learning rate. decrease your learning rate. And yes, increase the number of filters and make sure you are applying regularization. I dont know java May 12 '18 at 11:47
• What is usually a good learning rate for conv. nets? May 12 '18 at 11:48
• use adam and learning rate of 0.001-0.0001, beta1 = 0.9, beta2 = 0.999 May 12 '18 at 11:50

Training:

• 500 images to train on. (perfect. once you overfit this, you are on right track)
• N iterations with 5 images each time selected randomly and trained with a learning rate of 0.3 * overall_error. (hope you are not selecting randomly with replacement)

• Stochastic gradient descent with the backpropagation algorithm (use adam and learning rate of 0.001-0.0001, beta1 = 0.9, beta2 = 0.999)

Problems:

1. My network can learn up to 500 images perfectly but when I go higher than that, my accuracy goes down. (Probably not enough activation maps or sth.like that) (havent encountered this. Maybe some images are getting included where the gradient is exploding and network is dud after that. remember your learning rate was 0.3*overall_error)

2. When I take more input data, it might happen that my overall error reduces first, but ends up increasing towards infinity (learning rate too high. maybe the same problem images coming later on)

3. The filter of my first convolution layer look like this:

0,26756  0,3341   0,57849
-0,40428 0,10674  0,53553        (filter 1)
-0,27761 0,17577  -0,32525

177,92366 186,30331 -173,91031
-74,38991 1,29136  28,72685      (filter 5)
72,06218 37,42708 27,14726

.
.
.


Is it a normal thing that some filters have values close to zero and some other filters have values up to 1 million? (I had that aswell) (learning rate too high. typical)

4. If I use a ReLU activation function in the second convolution layer instead of the TanH activation function, my network error does not reduce at all and my accuracy is around 10% (on the Mnist dataset) -> pretty much random (relu dies. once it get knocked out of data cloud, nothing can bring it back. basically it actiavates 0 at each data point so no gradient ever goes back beyond to it and hence no updates are made)

• what do you mean with "randomly with replacement"? May 12 '18 at 12:14
• dont train on one image more than once each epoch. this leads to somewhat catastrophic interference. May 12 '18 at 12:20
• Oh no, I am just taking 10 random images. But no one will be used twice May 12 '18 at 12:21
• And I tested it with a lower learning rate and it solved everything. ReLU works. Everything works :)) Thank you very much May 12 '18 at 12:22
• That is the way to train. btw, randomize the dataset initially and consume it [0-32] [33-64] ... and so in in such batches. Batch size of power of 2 works best May 12 '18 at 12:22