Let's say we have binary classification problem: 0 vs 1. Some set of images needs to be mapped to those labels by convolutional neural network.

Main trend for now is to use ReLu as activation unit, nonlinearity, after convolution(or after conv + BN). ReLU cannot give negative values by definition(pReLU can, but let's stick with ReLU for now).

But if I use at the last layer (common practice for binary problem) sigmoid activation function, feeding it's output further to Loss function, then I think I have a problem: to get 1 with Sigmoid it's all OK I need positieve values. But to get 0 with Sigmoid I need negative argument, which is supressed by ReLU! And best I can have is 0.5 with 0 argument.

I have just encountered this problem, described above. But in most cases it works: VGG-16 uses ReLU, but you can get 0 value at the end if I'll use Sigmoid - I think due to fully connected layers that are able to produce negative values.

But here is the trick part: ResNet has no fully connected layers, average pooling at the end, it is fully convolutional with relus. But there I was able to recieve 0 value(with extra conv at the end without following Relu, didnt tried it without).

So did anyone thought it before - use of Sigmoid function at the end, as probability interpretation, or it is counterintuitive with Relu nets and if something is working this way - it is a miracle?

Should I stick with Softmax(I like this function less that Sigmoid, it less intuitive clear why it is a probability)?


1 Answer 1


You are forgetting about the weights. A Neural Network consist of activations (greater or equal to zero with the ReLu activation function) and weights.

The weights can be positive or negative. By weighting a positive ReLu output negatively a negative input for the next layer (e.g. a layer with a sigmoid activation function) emerges.

The possible weighted activations of a ReLu layer are thus $[-\infty, \infty]$. The sigmoid layer squashes (sum of) inputs between $[0,1]$ and can be used for a probabilistic interpretation. If you have multiple outputs you can use a softmax layer to normalize all your outputs to sum to 1.

  • $\begingroup$ I get what you mean. But I tried to mention it in my post: in fully convolutional networks after last Relu maximum 1-2 layers and how it can be enough to turn zeros obtained in previous layers to something negative? Here is the thought that it can be task specifiec: for classification that might work just fine, but I wrote it keeping in mind binary segmentation task. There is a lot of sparsity(zeros) in the output at last layers, and last convolutions gets multiplied by zeros in many cases. $\endgroup$ Jan 20, 2017 at 9:54
  • 1
    $\begingroup$ You don't turn zeros to something negative, you turn positive values and multiply them by negative weights to get a negative input to sigmoid. $\endgroup$
    – Lugi
    Mar 15, 2018 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.