How neural networks manages to learn 0 class for sigmoidal Cross-Entropy loss function using ReLU as activation unit?

Question

Let's say we have binary classification problem: 0 vs 1. Some set of images needs to be mapped to those labels by convolutional neural network.

Main trend for now is to use ReLu as activation unit, nonlinearity, after convolution(or after conv + BN). ReLU cannot give negative values by definition(pReLU can, but let's stick with ReLU for now).

But if I use at the last layer (common practice for binary problem) sigmoid activation function, feeding it's output further to Loss function, then I think I have a problem: to get 1 with Sigmoid it's all OK I need positieve values. But to get 0 with Sigmoid I need negative argument, which is supressed by ReLU! And best I can have is 0.5 with 0 argument.

I have just encountered this problem, described above. But in most cases it works: VGG-16 uses ReLU, but you can get 0 value at the end if I'll use Sigmoid - I think due to fully connected layers that are able to produce negative values.

But here is the trick part: ResNet has no fully connected layers, average pooling at the end, it is fully convolutional with relus. But there I was able to recieve 0 value(with extra conv at the end without following Relu, didnt tried it without).

So did anyone thought it before - use of Sigmoid function at the end, as probability interpretation, or it is counterintuitive with Relu nets and if something is working this way - it is a miracle?

Should I stick with Softmax(I like this function less that Sigmoid, it less intuitive clear why it is a probability)?

Answer 1

You are forgetting about the weights. A Neural Network consist of activations (greater or equal to zero with the ReLu activation function) and weights.

The weights can be positive or negative. By weighting a positive ReLu output negatively a negative input for the next layer (e.g. a layer with a sigmoid activation function) emerges.

The possible weighted activations of a ReLu layer are thus $[-\infty, \infty]$. The sigmoid layer squashes (sum of) inputs between $[0,1]$ and can be used for a probabilistic interpretation. If you have multiple outputs you can use a softmax layer to normalize all your outputs to sum to 1.