In Keras and tensorflow there is a loss function called binary crossentropy. When i use that the output neurons will only produce 0 and 1.

I see that in the code of keras, binary cross entropy is linked to sigmoid_cross_entropy_with_logits in tensorflow, and from there I assume it goes on to a c++ implementation.

I'd like to know how exactly is it decided whether it's 0 or 1. Is it simply rounding the output? I would like to know because I want create a custom loss function which will have weights and want to make sure I'm not introducing any bias.


closed as off-topic by Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, John Dec 21 '17 at 22:01

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, kjetil b halvorsen, Stephan Kolassa, mdewey, John
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ this is definitely not off topic $\endgroup$ – Nickpick Feb 13 '18 at 10:42

When i use that the output neurons will only produce 0 and 1.

Cross-entropy produces scores in $[0,\infty)$.

Examining the expression for cross-entropy should make this clear. For model parameters $\theta$, labels $y$ and predicted probabilities $p_i$, it is: $$ \mathcal{L}(\theta)= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i)\log(1-p_i) \right] $$ The function sigmoid_cross_entropy_loss_with_logits is evaluating an equivalent expression that takes $\text{logit}(p_i)$ as inputs; this can be numerically nicer. (Working though the algebra of the above equation will provide you with the exact expression that the function sigmoid_cross_entropy_loss_with_logits is evaluating.)

All of this to say: the output of the function sigmoid_cross_entropy_loss_with_logits cannot possibly be binary for all outputs. Are you post-processing the probabilities or logits in some way? It is very common for people to employ argmax to coerce outcomes to be binary, 1-hot representations of class membership.

To answer the titular question, binary cross entropy loss doesn’t decide the outputs. It just scores “how wrong” the model is; larger values imply the model is “more wrong.”

  • $\begingroup$ Thanks. So the loss function is independent of the last layer? Or are there any criteria that need to be considered? $\endgroup$ – Nickpick Dec 8 '17 at 20:06
  • $\begingroup$ I actually wanted to know how the output of keras’ binary cross entropy is coerced to binary. Is it just using a threshold? $\endgroup$ – Nickpick Dec 11 '17 at 8:53
  • $\begingroup$ @Nickpick It doesn't coerce to binary. The only way it's producing outputs in $\{0, 1\}$ is if all inputs are either $e$ or $1$. The function just calls the corresponding TensorFlow function and returns the result. github.com/anayebi/keras-extra/blob/master/… So the coercion to binary must happen somewhere else. You should find an appropriate forum for debugging Keras code and ask your code question there. $\endgroup$ – Sycorax Dec 11 '17 at 15:42
  • $\begingroup$ Ok makes sense. I found it now: def binary_accuracy(y_true, y_pred): return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1) It simply rounds the value. $\endgroup$ – Nickpick Dec 11 '17 at 16:06
  • $\begingroup$ @Nickpick Yep, it looks like you were applying a post-processing to the network outputs! $\endgroup$ – Sycorax Dec 11 '17 at 16:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.