CNN: Range of filters and activation functions

Question

I've been trying to code a neural network library in Java which contains very simple stuff like a Convolution-layer, Dense-Layer and a Flattening-Layer.

I am pretty sure that my Dense-layer works as expected but I am not sure about the convolution. My network architecture looks like this:

    NetworkBuilder builder = new NetworkBuilder(1, 28, 28);
    builder.addLayer(new ConvLayer(8, 3, 1, 2)   //act.-maps, filter_size, stride, padding
            .biasRange(0, 0)
            .weightsRange(-2, 2)
            .setActivationFunction(new ReLU()));
    builder.addLayer(new PoolingLayer(2));       //downsampling factor
    builder.addLayer(new ConvLayer(26, 5, 1, 0)  //act.-maps, filter_size, stride, padding
            .biasRange(0, 0)
            .weightsRange(-2, 2)
            .setActivationFunction(new TanH()));
    builder.addLayer(new PoolingLayer(2));       //downsampling factor
    builder.addLayer(new TransformationLayer()); //Flattenig (3D -> 1D)
    builder.addLayer(new DenseLayer(120)         //120 neurons
            .setActivationFunction(new Sigmoid())
    );
    builder.addLayer(new DenseLayer(10)
            .setActivationFunction(new Softmax())
    );
    Network network = builder.buildNetwork();
    network.setErrorFunction(new CrossEntropy());

Training:

• 500 images to train on.

• N iterations with 5 images each time selected randomly and trained with a learning rate of 0.3 * overall_error.

• Stochastic gradient descent with the backpropagation algorithm

Problems:

1. My network can learn up to 500 images perfectly but when I go higher than that, my accuracy goes down. (Probably not enough activation maps or sth.like that)

2. When I take more input data, it might happen that my overall error reduces first, but ends up increasing towards infinity

3. The filter of my first convolution layer look like this:

   0,26756  0,3341   0,57849         
   -0,40428 0,10674  0,53553        (filter 1)     
   -0,27761 0,17577  -0,32525  
   
   177,92366 186,30331 -173,91031        
   -74,38991 1,29136  28,72685      (filter 5)   
   72,06218 37,42708 27,14726  
   
       .
       .
       .
   

   Is it a normal thing that some filters have values close to zero and some other filters have values up to 1 million? (I had that aswell)

4. If I use a ReLU activation function in the second convolution layer instead of the TanH activation function, my network error does not reduce at all and my accuracy is around 10% (on the Mnist dataset) -> pretty much random

If those things that I mentioned above are things that can sometimes happen in neural networks, I assume my implementation is correct but my training parameters are wrong but if not, I need to check my code again.

Answer 1

Training:

• 500 images to train on. (perfect. once you overfit this, you are on right track)

• N iterations with 5 images each time selected randomly and trained with a learning rate of 0.3 * overall_error. (hope you are not selecting randomly with replacement)

• Stochastic gradient descent with the backpropagation algorithm (use adam and learning rate of 0.001-0.0001, beta1 = 0.9, beta2 = 0.999)

Problems:

1. My network can learn up to 500 images perfectly but when I go higher than that, my accuracy goes down. (Probably not enough activation maps or sth.like that) (havent encountered this. Maybe some images are getting included where the gradient is exploding and network is dud after that. remember your learning rate was 0.3*overall_error)

2. When I take more input data, it might happen that my overall error reduces first, but ends up increasing towards infinity (learning rate too high. maybe the same problem images coming later on)

3. The filter of my first convolution layer look like this:

   0,26756  0,3341   0,57849         
   -0,40428 0,10674  0,53553        (filter 1)     
   -0,27761 0,17577  -0,32525  
   
   177,92366 186,30331 -173,91031        
   -74,38991 1,29136  28,72685      (filter 5)   
   72,06218 37,42708 27,14726  
   
       .
       .
       .
   

   Is it a normal thing that some filters have values close to zero and some other filters have values up to 1 million? (I had that aswell) (learning rate too high. typical)

4. If I use a ReLU activation function in the second convolution layer instead of the TanH activation function, my network error does not reduce at all and my accuracy is around 10% (on the Mnist dataset) -> pretty much random (relu dies. once it get knocked out of data cloud, nothing can bring it back. basically it actiavates 0 at each data point so no gradient ever goes back beyond to it and hence no updates are made)