# 1 neuron BCE loss VS 2 neurons CE loss

I built a custom version of YOLO that should only detect one type of objects, where the objectness measure (which tells how likely a bounding-box contains an object of any type), is learned using a logistic regression (BCE loss), with a single neuron. Since I have only one type of object, objectness is the only measure that I use at inference.

My problem is that this neuron almost only has extreme outputs. Fed to the Sigmoid function $$\frac{1}{1+e^{-x}}$$, the values of $$x$$ are either very low (negative with high abs value) or positive very high, resulting in almost binary outputs, 0 and 1. This is very similar to a results that I got with MSE loss on a single neuron, so what is the benefit of using a logistic regression?

Is it related to the nature of the BCE loss? Should I get objectness scores that are spread all over the range if I use CE with softmax on 2 neurons, one for negative objectness and one for positive?

• @Sycorax thank you. It doesn't actually answer my question. I know to tell the difference between BCE and CE and when each should be used. My question is regarding BCE causing almost only outputs at the extremes. Sep 1 '20 at 14:06

# Cross-entropy penalizes predictions that are far from the label.

My problem is that this neuron almost only has extreme outputs. Fed to the Sigmoid function $$p=\frac{1}{1+e^{-x}}$$, the values of 𝑥 are either very low (negative with high abs value) or positive very high, resulting in almost binary outputs, 0 and 1. Is it related to the nature of the BCE loss?

Yes. The cross-entropy loss $$L=y\log(p)-(1-y)\log(1-p)$$ for $$p\in[0,1]$$ is minimized at zero. It achieves the value of zero in two cases:

1. If $$y=1$$, then $$L$$ is minimized when $$p=1$$.
2. If $$y=0$$, then $$L$$ is minimized when $$p=0$$.

Since $$\log(x)$$ is monotonic increasing, this also works the other way around: $$L$$ is large when $$p$$ is very far from the true value $$y$$.

We know that $$0 \le f(x)_i \le 1$$, so the $$f(x)_i$$ that is farthest from the extreme values of 0 and 1 is $$f(x)_1=f(x)_2=\frac{1}{2}$$. This occurs when $$x_1=x_2$$. For $$y=1$$, the loss $$L$$ for $$p=\frac{1}{2}$$ is larger than the loss $$L$$ for any $$p>\frac{1}{2}$$; likewise, for $$y=0$$, the loss $$L$$ for $$p=\frac{1}{2}$$ is larger than the loss $$L$$ for any $$p < \frac{1}{2}$$. So minimizing the loss will push $$p$$ to be closer to the labels $$y$$. (This may not always be possible; in that case, the network will not achieve $$p=1$$ for all $$y=1$$ or $$p=0$$ for all $$y=0$$. The simplest case where this happens is when the same input has opposite labels. However, OP's particular model is achieving values near 0 and 1 on this data, so we know that predicting $$\frac{1}{2}$$ for everything has a larger loss in this particular case.)

In other words, small loss values imply extreme predictions and vice versa.

This should hint at why cross-entropy loss tends to produce extreme predictions: the model assigns extremely steep penalties to any predictions that are far from $$y$$. Neural networks are especially susceptible to this phenomenon, because a neural network tends to be over-parameterized. These additional degrees of freedom can allow the network to find solutions which find very small loss values.

## Additionally, cross entropy loss is steeper and assigns larger penalties to very incorrect predictions.

This is very similar to a results that I got with MSE loss on a single neuron, so what is the benefit of using a logistic regression?

The values of MSE loss are bounded in $$[0,1]$$. The gradient of MSE loss is $$2(y-p)$$, so the largest value of the gradient is 2.

The values of cross-entropy loss is bounded below by 0, but increases without bound. The gradient of cross-entropy loss is $$\frac{p-y}{p-p^2}$$, which is very steep for $$p$$ far from $$y$$.

Indeed, for $$y=1$$, the cross-entropy gradient is larger than the MSE gradient whenever $$p < \exp(-2)$$; and likewise, for $$y=0$$, the cross-entropy gradient is larger than the MSE gradient whenever $$p > 1 - \exp(-2)$$. The steepness of the cross-entropy gradient is nonlinear in $$p$$, so using a larger, fixed learning rate does not eliminate this difference between the MSE and CE models.

Should I get objectness scores that are spread all over the range if I use CE with softmax on 2 neurons, one for negative objectness and one for positive?

# Changing the loss function makes no difference.

OP describes a model of binary events $$y\in\{0,1\}$$. A neural network with a single output predicts the event $$y=1$$ with probability $$p\in[0,1]$$. Using a Kolmogorov axiom, we know tha the probability of the mutually exclusive event $$y=0$$ is $$1-p$$.

The binary cross entropy loss of this prediction is $$L=-y\log(p)-(1-y)\log(1-p)$$.

Now consider a neural network model that has two output neurons, such that the outputs are non-negative and sum to 1. The first neuron predicts a value $$p$$ and the second neuron predicts $$1-p$$.

The cross entropy loss of this prediction is $$L=-y\log(p)-(1-y)\log(1-p),$$ exactly identical to the case of a single output neuron. This is true regardless of what activation function we use to come up with the values $$p$$ and $$1-p$$, as long as that activation returns $$p\in[0,1]$$.

Clearly, the choice of binary cross entropy or categorical cross entropy for the case of two classes makes no difference because both formulations give the same loss $$L$$.

# Choosing softmax makes no difference.

So now we need to decide whether or not softmax makes any difference.

If the softmax function is used to obtain $$p, 1-p$$ from the two output neurons, then we have two distinct inputs to the softmax function, $$x_1, x_2$$:

$$f(x)_i = \frac{\exp(x_i)}{\exp(x_1)+\exp(x_2)}.$$

If we use the same weights and biases as the original model, we will compute the same $$x$$. Applying a certain affine layer and softmax to $$x$$ will give identical predictions, and therefore have identical loss.

\begin{align} p&=\frac{1}{1+\exp(-x)} \\ &=\frac{\exp(0x+0)}{\exp(0x+0)+\exp(-1x+0)}\\ &=\frac{\exp(a_1x+b_1)}{\exp(a_1x+b_1)+\exp(a_2x+b_2)} \end{align}

Presumably, the network alternates between affine layers and nonlinear activation functions. Because the step applied to $$x$$ was an activation function, we know that the step before $$x$$ was a linear operation. So the above demonstration doesn't even require any additional layers because linear functions are closed under composition.

• Thank you so much for your so elaborate answer. It may take me some time to understand and realize everything. Thank you so much again! Sep 1 '20 at 16:51
• Everything is very clear and very helpful. Thank you so much! Regarding the softmax part, which was harder for me to understand - does the claim say that BCE can be simulated by it using a certain assignment, or does it say that it must be identical? Sep 1 '20 at 17:07
• It just shows that there exists a softmax layer which has the same mapping from $x$ to $p$. If we further suppose that the sigmoid model achieves a minimum loss $L$, then the softmax model which achieves the minimum loss will be identical to the sigmoid model.
– Sycorax
Sep 1 '20 at 17:15
• When you do softmax, if the GT is positive, then your loss is e^p / (e^p + e^n). If you divide both numerator and denominator by e^p, you get 1 / (1 + e^(n-p)). So it is almost like BCE with Sigmoid on a single neuron, but it also includes another value, n, which is the activation of the neuron that represents the negative class. Isn't it going to behave differently from a single neuron with BCE? I think it is only identical to BCE if you know that n is always zero in CE with two neurons. Sep 9 '20 at 11:41
• One way to make n always zero is to multiply it by zero, which is what my answer shows.
– Sycorax
Sep 9 '20 at 13:28