I have two CNN versions which are distinguished by a sigmoid layer.
- CNN | last two layers:
- CNN | last layer:
My output range of my ground truth values is
The loss function I use is the
When I train both networks the second one outperforms the first one by far.
For example: 1. At the beginning: loss = 230 1. After 3 epochs: loss = 23
- At the beginning: loss = 18
- After 100 iterations loss = 4
I do not understand why the version with the
SIGMOID does never get near the solution without the sigmoid. I have been reading up on this and some people say if the
L2 loss does not go well with the
SIGMOID which can be proven mathematically. However, in the end, I would understand if there is some sort of difference for the loss, but the difference is huge.