1 neuron BCE loss VS 2 neurons CE loss

I built a custom version of YOLO that should only detect one type of objects, where the objectness measure (which tells how likely a bounding-box contains an object of any type), is learned using a logistic regression (BCE loss), with a single neuron. Since I have only one type of object, objectness is the only measure that I use at inference.

My problem is that this neuron almost only has extreme outputs. Fed to the Sigmoid function $$\frac{1}{1+e^{-x}}$$, the values of $$x$$ are either very low (negative with high abs value) or positive very high, resulting in almost binary outputs, 0 and 1. This is very similar to a results that I got with MSE loss on a single neuron, so what is the benefit of using a logistic regression?

Is it related to the nature of the BCE loss? Should I get objectness scores that are spread all over the range if I use CE with softmax on 2 neurons, one for negative objectness and one for positive?

• @Sycorax thank you. It doesn't actually answer my question. I know to tell the difference between BCE and CE and when each should be used. My question is regarding BCE causing almost only outputs at the extremes. – SomethingSomething Sep 1 '20 at 14:06

Cross-entropy penalizes predictions that are far from the label.

My problem is that this neuron almost only has extreme outputs. Fed to the Sigmoid function $$p=\frac{1}{1+e^{-x}}$$, the values of 𝑥 are either very low (negative with high abs value) or positive very high, resulting in almost binary outputs, 0 and 1. Is it related to the nature of the BCE loss?

Yes. The cross-entropy loss $$L=y\log(p)-(1-y)\log(1-p)$$ for $$p\in[0,1]$$ is minimized at zero. It achieves the value of zero in two cases:

1. If $$y=1$$, then $$L$$ is minimized when $$p=1$$.
2. If $$y=0$$, then $$L$$ is minimized when $$p=0$$.

Since $$\log(x)$$ is monotonic increasing, this also works the other way around: $$L$$ is large when $$p$$ is very far from the true value $$y$$.

We know that $$0 \le f(x)_i \le 1$$, so the $$f(x)_i$$ that is farthest from the extreme values of 0 and 1 is $$f(x)_1=f(x)_2=\frac{1}{2}$$. This occurs when $$x_1=x_2$$. For $$y=1$$, the loss $$L$$ for $$p=\frac{1}{2}$$ is larger than the loss $$L$$ for any $$p>\frac{1}{2}$$; likewise, for $$y=0$$, the loss $$L$$ for $$p=\frac{1}{2}$$ is larger than the loss $$L$$ for any $$p < \frac{1}{2}$$. So minimizing the loss will push $$p$$ to be closer to the labels $$y$$. (This may not always be possible; in that case, the network will not achieve $$p=1$$ for all $$y=1$$ or $$p=0$$ for all $$y=0$$. The simplest case where this happens is when the same input has opposite labels. However, OP's particular model is achieving values near 0 and 1 on this data, so we know that predicting $$\frac{1}{2}$$ for everything has a larger loss in this particular case.)

In other words, small loss values imply extreme predictions and vice versa.

This should hint at why cross-entropy loss tends to produce extreme predictions: the model assigns extremely steep penalties to any predictions that are far from $$y$$. Neural networks are especially susceptible to this phenomenon, because a neural network tends to be over-parameterized. These additional degrees of freedom can allow the network to find solutions which find very small loss values.

Additionally, cross entropy loss is steeper and assigns larger penalties to very incorrect predictions.

This is very similar to a results that I got with MSE loss on a single neuron, so what is the benefit of using a logistic regression?

The values of MSE loss are bounded in $$[0,1]$$. The gradient of MSE loss is $$2(y-p)$$, so the largest value of the gradient is 2.

The values of cross-entropy loss is bounded below by 0, but increases without bound. The gradient of cross-entropy loss is $$\frac{p-y}{p-p^2}$$, which is very steep for $$p$$ far from $$y$$.

Indeed, for $$y=1$$, the cross-entropy gradient is larger than the MSE gradient whenever $$p < \exp(-2)$$; and likewise, for $$y=0$$, the cross-entropy gradient is larger than the MSE gradient whenever $$p > 1 - \exp(-2)$$. The steepness of the cross-entropy gradient is nonlinear in $$p$$, so using a larger, fixed learning rate does not eliminate this difference between the MSE and CE models.

Should I get objectness scores that are spread all over the range if I use CE with softmax on 2 neurons, one for negative objectness and one for positive?

Changing the loss function makes no difference.

OP describes a model of binary events $$y\in\{0,1\}$$. A neural network with a single output predicts the event $$y=1$$ with probability $$p\in[0,1]$$. Using a Kolmogorov axiom, we know tha the probability of the mutually exclusive event $$y=0$$ is $$1-p$$.

The binary cross entropy loss of this prediction is $$L=-y\log(p)-(1-y)\log(1-p)$$.

Now consider a neural network model that has two output neurons, such that the outputs are non-negative and sum to 1. The first neuron predicts a value $$p$$ and the second neuron predicts $$1-p$$.

The cross entropy loss of this prediction is $$L=-y\log(p)-(1-y)\log(1-p),$$ exactly identical to the case of a single output neuron. This is true regardless of what activation function we use to come up with the values $$p$$ and $$1-p$$, as long as that activation returns $$p\in[0,1]$$.

Clearly, the choice of binary cross entropy or categorical cross entropy for the case of two classes makes no difference because both formulations give the same loss $$L$$.

Choosing softmax makes no difference.

So now we need to decide whether or not softmax makes any difference.

If the softmax function is used to obtain $$p, 1-p$$ from the two output neurons, then we have two distinct inputs to the softmax function, $$x_1, x_2$$:

$$f(x)_i = \frac{\exp(x_i)}{\exp(x_1)+\exp(x_2)}.$$

If we use the same weights and biases as the original model, we will compute the same $$x$$. Applying a certain affine layer and softmax to $$x$$ will give identical predictions, and therefore have identical loss.

\begin{align} p&=\frac{1}{1+\exp(-x)} \\ &=\frac{\exp(0x+0)}{\exp(0x+0)+\exp(-1x+0)}\\ &=\frac{\exp(a_1x+b_1)}{\exp(a_1x+b_1)+\exp(a_2x+b_2)} \end{align}

Presumably, the network alternates between affine layers and nonlinear activation functions. Because the step applied to $$x$$ was an activation function, we know that the step before $$x$$ was a linear operation. So the above demonstration doesn't even require any additional layers because linear functions are closed under composition.

• Thank you so much for your so elaborate answer. It may take me some time to understand and realize everything. Thank you so much again! – SomethingSomething Sep 1 '20 at 16:51
• Everything is very clear and very helpful. Thank you so much! Regarding the softmax part, which was harder for me to understand - does the claim say that BCE can be simulated by it using a certain assignment, or does it say that it must be identical? – SomethingSomething Sep 1 '20 at 17:07
• It just shows that there exists a softmax layer which has the same mapping from $x$ to $p$. If we further suppose that the sigmoid model achieves a minimum loss $L$, then the softmax model which achieves the minimum loss will be identical to the sigmoid model. – Sycorax Sep 1 '20 at 17:15
• When you do softmax, if the GT is positive, then your loss is e^p / (e^p + e^n). If you divide both numerator and denominator by e^p, you get 1 / (1 + e^(n-p)). So it is almost like BCE with Sigmoid on a single neuron, but it also includes another value, n, which is the activation of the neuron that represents the negative class. Isn't it going to behave differently from a single neuron with BCE? I think it is only identical to BCE if you know that n is always zero in CE with two neurons. – SomethingSomething Sep 9 '20 at 11:41
• One way to make n always zero is to multiply it by zero, which is what my answer shows. – Sycorax Sep 9 '20 at 13:28