Cross-entropy penalizes predictions that are far from the label.
My problem is that this neuron almost only has extreme outputs. Fed to the Sigmoid function $p=\frac{1}{1+e^{-x}}$, the values of 𝑥 are either very low (negative with high abs value) or positive very high, resulting in almost binary outputs, 0 and 1. Is it related to the nature of the BCE loss?
Yes. The cross-entropy loss $L=y\log(p)-(1-y)\log(1-p)$ for $p\in[0,1]$ is minimized at zero. It achieves the value of zero in two cases:
- If $y=1$, then $L$ is minimized when $p=1$.
- If $y=0$, then $L$ is minimized when $p=0$.
Since $\log(x)$ is monotonic increasing, this also works the other way around: $L$ is large when $p$ is very far from the true value $y$.
We know that $0 \le f(x)_i \le 1$, so the $f(x)_i$ that is farthest from the extreme values of 0 and 1 is $f(x)_1=f(x)_2=\frac{1}{2}$. This occurs when $x_1=x_2$. For $y=1$, the loss $L$ for $p=\frac{1}{2}$ is larger than the loss $L$ for any $p>\frac{1}{2}$; likewise, for $y=0$, the loss $L$ for $p=\frac{1}{2}$ is larger than the loss $L$ for any $p < \frac{1}{2}$. So minimizing the loss will push $p$ to be closer to the labels $y$. (This may not always be possible; in that case, the network will not achieve $p=1$ for all $y=1$ or $p=0$ for all $y=0$. The simplest case where this happens is when the same input has opposite labels. However, OP's particular model is achieving values near 0 and 1 on this data, so we know that predicting $\frac{1}{2}$ for everything has a larger loss in this particular case.)
In other words, small loss values imply extreme predictions and vice versa.
This should hint at why cross-entropy loss tends to produce extreme predictions: the model assigns extremely steep penalties to any predictions that are far from $y$. Neural networks are especially susceptible to this phenomenon, because a neural network tends to be over-parameterized. These additional degrees of freedom can allow the network to find solutions which find very small loss values.
Additionally, cross entropy loss is steeper and assigns larger penalties to very incorrect predictions.
This is very similar to a results that I got with MSE loss on a single neuron, so what is the benefit of using a logistic regression?
The values of MSE loss are bounded in $[0,1]$. The gradient of MSE loss is $2(y-p)$, so the largest value of the gradient is 2.
The values of cross-entropy loss is bounded below by 0, but increases without bound. The gradient of cross-entropy loss is $\frac{p-y}{p-p^2}$, which is very steep for $p$ far from $y$.
Indeed, for $y=1$, the cross-entropy gradient is larger than the MSE gradient whenever $p < \exp(-2)$; and likewise, for $y=0$, the cross-entropy gradient is larger than the MSE gradient whenever $p > 1 - \exp(-2)$. The steepness of the cross-entropy gradient is nonlinear in $p$, so using a larger, fixed learning rate does not eliminate this difference between the MSE and CE models.
Should I get objectness scores that are spread all over the range if I use CE with softmax on 2 neurons, one for negative objectness and one for positive?
Changing the loss function makes no difference.
OP describes a model of binary events $y\in\{0,1\}$. A neural network with a single output predicts the event $y=1$ with probability $p\in[0,1]$. Using a Kolmogorov axiom, we know tha the probability of the mutually exclusive event $y=0$ is $1-p$.
The binary cross entropy loss of this prediction is $L=-y\log(p)-(1-y)\log(1-p)$.
Now consider a neural network model that has two output neurons, such that the outputs are non-negative and sum to 1. The first neuron predicts a value $p$ and the second neuron predicts $1-p$.
The cross entropy loss of this prediction is
$L=-y\log(p)-(1-y)\log(1-p),$ exactly identical to the case of a single output neuron. This is true regardless of what activation function we use to come up with the values $p$ and $1-p$, as long as that activation returns $p\in[0,1]$.
Clearly, the choice of binary cross entropy or categorical cross entropy for the case of two classes makes no difference because both formulations give the same loss $L$.
Choosing softmax makes no difference.
So now we need to decide whether or not softmax makes any difference.
If the softmax function is used to obtain $p, 1-p$ from the two output neurons, then we have two distinct inputs to the softmax function, $x_1, x_2$:
$$f(x)_i = \frac{\exp(x_i)}{\exp(x_1)+\exp(x_2)}.$$
If we use the same weights and biases as the original model, we will compute the same $x$. Applying a certain affine layer and softmax to $x$ will give identical predictions, and therefore have identical loss.
$$\begin{align}
p&=\frac{1}{1+\exp(-x)} \\
&=\frac{\exp(0x+0)}{\exp(0x+0)+\exp(-1x+0)}\\
&=\frac{\exp(a_1x+b_1)}{\exp(a_1x+b_1)+\exp(a_2x+b_2)}
\end{align}
$$
Presumably, the network alternates between affine layers and nonlinear activation functions. Because the step applied to $x$ was an activation function, we know that the step before $x$ was a linear operation. So the above demonstration doesn't even require any additional layers because linear functions are closed under composition.
See also: Neural Network: For Binary Classification use 1 or 2 output neurons?
