How does binary-crossentropy decide the output [closed]

In Keras and tensorflow there is a loss function called binary crossentropy. When i use that the output neurons will only produce 0 and 1.

I see that in the code of keras, binary cross entropy is linked to sigmoid_cross_entropy_with_logits in tensorflow, and from there I assume it goes on to a c++ implementation.

I'd like to know how exactly is it decided whether it's 0 or 1. Is it simply rounding the output? I would like to know because I want create a custom loss function which will have weights and want to make sure I'm not introducing any bias.

closed as off-topic by Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, JohnDec 21 '17 at 22:01

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, John
If this question can be reworded to fit the rules in the help center, please edit the question.

• this is definitely not off topic – Nickpick Feb 13 '18 at 10:42

When i use that the output neurons will only produce 0 and 1.

Cross-entropy produces scores in $[0,\infty)$.

Examining the expression for cross-entropy should make this clear. For model parameters $\theta$, labels $y$ and predicted probabilities $p_i$, it is: $$\mathcal{L}(\theta)= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i)\log(1-p_i) \right]$$ The function sigmoid_cross_entropy_loss_with_logits is evaluating an equivalent expression that takes $\text{logit}(p_i)$ as inputs; this can be numerically nicer. (Working though the algebra of the above equation will provide you with the exact expression that the function sigmoid_cross_entropy_loss_with_logits is evaluating.)

All of this to say: the output of the function sigmoid_cross_entropy_loss_with_logits cannot possibly be binary for all outputs. Are you post-processing the probabilities or logits in some way? It is very common for people to employ argmax to coerce outcomes to be binary, 1-hot representations of class membership.

To answer the titular question, binary cross entropy loss doesn’t decide the outputs. It just scores “how wrong” the model is; larger values imply the model is “more wrong.”

• Thanks. So the loss function is independent of the last layer? Or are there any criteria that need to be considered? – Nickpick Dec 8 '17 at 20:06
• I actually wanted to know how the output of keras’ binary cross entropy is coerced to binary. Is it just using a threshold? – Nickpick Dec 11 '17 at 8:53
• @Nickpick It doesn't coerce to binary. The only way it's producing outputs in $\{0, 1\}$ is if all inputs are either $e$ or $1$. The function just calls the corresponding TensorFlow function and returns the result. github.com/anayebi/keras-extra/blob/master/… So the coercion to binary must happen somewhere else. You should find an appropriate forum for debugging Keras code and ask your code question there. – Sycorax Dec 11 '17 at 15:42
• Ok makes sense. I found it now: def binary_accuracy(y_true, y_pred): return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1) It simply rounds the value. – Nickpick Dec 11 '17 at 16:06
• @Nickpick Yep, it looks like you were applying a post-processing to the network outputs! – Sycorax Dec 11 '17 at 16:34