In the past, I've worked with the MNIST dataset and I am currently working with Java. I do not use any external libraries like numpy (for python) or something like that. I started from scratch and build a feed forward network. I mastered the backpropagation algorithm and managed to get a result of 94%. Obviously, I wanted to get something higher than 94% so I started building and using a CNN. My problem now is that there exist a couple of different explanations of each layer and I am not completely sure which one is the "best" or most efficient way (if there is one).
Well. The feed forward in every Layer worked fine. My convolution is working fine but here I come to my first question:
1) Normally a ReLU would be added after my conv. layer. I've seen a couple of different functions that can be used here. I am familiar with the sigmoid function that I use in my FC-Layer. Most people like to use the max(x,0) function here but I don't understand how to backpropagate this. So which function shall I use ? Wouldn't it be more efficient to handle ReLU as just a function that I add into my Conv-Layer instead of handling it as it's own Layer ?
2) When I calculate the error signals through my pooling layer I would set the error signal of the neuron with the highest activation value to the error signal I just calculated previously. My problem now is that I've read that most people set the error signal to 1 and everything else to 0. Why would you do this ?? I've added a picture for illustration. (My Paint-Skills are not the best :)) Pooling Layer painted in Paint with error signals
3) Coming back to the Convolution Layer and calculating the signal errors. I thought I could use the same formula like in my fully connected layer:
//i = layer we are currently working in
for (j = 0; j < outputs[i].length; j++) {
Sum = 0;
for (k = 0; k < outputs[i + 1].length; k++) {
Sum = Sum + weights[i + 1][k][j] * error_signals[i + 1][k];
}
error_signals[i][j] = derivatives[i][j] * Sum;
}
This works fine for my FC-Layer. but does this also work for my conv. layer ? And when I update my weights and biases. How can I do this ? I was thinking of updating each weight by the same formula as in the FC-Layer. That would also mean that I use the sigmoid function in my ReLU. The sharing weights stuff would just result in updating one weight more often each time I run my backpropagation.
Because I am just a "simple" student and not someone who has studied anything (not yet) I do not yet understand the math that is used here. I managed to handle the backpropagation in feed forward nets but still I do not completely understand why it works. I would be very happy if someone could tell me if my ideas work and if they do not work, tell me a better way (in a simple way please) :).
I am thankful for any help :)
