Let's say we have binary classification problem: 0 vs 1. Some set of images needs to be mapped to those labels by convolutional neural network.
Main trend for now is to use ReLu as activation unit, nonlinearity, after convolution(or after conv + BN). ReLU cannot give negative values by definition(pReLU can, but let's stick with ReLU for now).
But if I use at the last layer (common practice for binary problem) sigmoid activation function, feeding it's output further to Loss function, then I think I have a problem: to get 1 with Sigmoid it's all OK I need positieve values. But to get 0 with Sigmoid I need negative argument, which is supressed by ReLU! And best I can have is 0.5 with 0 argument.
I have just encountered this problem, described above. But in most cases it works: VGG-16 uses ReLU, but you can get 0 value at the end if I'll use Sigmoid - I think due to fully connected layers that are able to produce negative values.
But here is the trick part: ResNet has no fully connected layers, average pooling at the end, it is fully convolutional with relus. But there I was able to recieve 0 value(with extra conv at the end without following Relu, didnt tried it without).
So did anyone thought it before - use of Sigmoid function at the end, as probability interpretation, or it is counterintuitive with Relu nets and if something is working this way - it is a miracle?
Should I stick with Softmax(I like this function less that Sigmoid, it less intuitive clear why it is a probability)?