# LogLoss in neural networks - Binary

I recently presented my M.D work and one question got me into trouble.

Context:

In my neural network classification, I use 1 single output sigmoid neuron.

My problem requires that i classify the input into two groups : 0 or 1.

For the loss function, I use lasagne.objectives.binary_crossentropy :

Computes the binary cross-entropy between predictions and targets.

L= $−tlog(p)− (1−t) log(1−p)$

Question made in my presentation by a teacher

"You cant be using binary cross-entropy because entropy is just used when your data either has noises or some probability, why and how do you use that loss function? You cant use it because for each input the model already classify it to either 1 group, so 'entropy' makes no sense".

Im confused, especially by the last part, because my model has a sigmoid output that ranges from 0 to 1. I know that the output from the sigmoid neuron isnt exactly a probability, However his question doesnt make sense to me.

Would someone help me come up with an answer for that?

Sources:

• Is your output perfectly represented by your input? If you have several different outputs for same input then you have some uncertainty (some call it noise, but I don't like that terminology) incorporated. In this case there is a distribution involved. – Cagdas Ozgenc Sep 28 '17 at 6:01
• Its the same output for the same input. – KenobiShan Sep 28 '17 at 12:36
• It seems like there was a miscommunication. I think you claimed that after the probabilistic output of your model you classified it as a zero or one given some threshold. However the log loss applies to your model before the threshold classification is applied, so it's calculated on the probabilistic outputs. I think your professor thought that you claimed that the $0$ or $1$ was the direct output of your model, to which cross-entropy cannot be applied. (If you wonder why this is, just substitute the values $0$ or $1$ in the cross-entropy expression and see what you get.) – Bridgeburners Sep 29 '17 at 12:46

As you said in the comments, (at least in the training set) there are no conflicting patterns. Thus, it is a little unusual to talk about cross entropy when there is no confusion (entropy) in the data. A more natural choice could have been using a hinge loss (as SVMs do). On the other hand, Logistic Regression (LR) may also yield good results even if you do not expect a distribution on $P(y|X)$. In fact, results from LR may be useful if you need some sort of "confidence" on your classification. It is easier to interpret a probability than a distance to a hyperplane in a large dimensional space (in the case of the hinge loss). Or you may desire to expand your results beyond binary classification and LR can do this naturally. Without being familiar with your work it is impossible for us Internet strangers to guess for what reasons you chose cross entropy for your problem.