1
$\begingroup$

Binary cross entropy is written as follows:

\begin{equation} \mathcal{L} = -y\log\left(\hat{y}\right)-(1-y)\log\left(1-\hat{y}\right) \end{equation}

In every reference that I read, when using binary cross entropy, they use labels 0 and 1, with activation the output layer is sigmoid. I wonder if it is possible to use cross entropy labeled -1 and 1 with the output layer using tanh activation?

$\endgroup$
2

1 Answer 1

1
$\begingroup$

No, you can’t. What would $\log\left(\hat{y}\right)$ be when $\hat{y}$ is (close to) -1?

There are simple workarounds. You can rescale your outputs to $[0, 1]$, or you can use Brier score instead of cross entropy, but why would you?

Using $\tanh$ activation functions in hidden layers is a natural thing to do, but in the output layer one advantage of sigmoid is that it has a natural probabilistic interpretation. As a consequence, the outputs are compatible with the cross entropy function you defined above.

$\endgroup$
3
  • $\begingroup$ But one could re-express OP’s loss function in a way that gives equivalent results for -1/+1 labels. $\endgroup$
    – Sycorax
    Commented May 26, 2023 at 10:34
  • $\begingroup$ @Sycorax Yep! $\endgroup$
    – Dave
    Commented May 26, 2023 at 10:52
  • $\begingroup$ Absolutely, but then it’s not (conceptually) cross-entropy anymore. I did mention rescaling the outputs. You could rescale the loss function instead. $\endgroup$ Commented May 26, 2023 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.