LogLoss in neural networks - Binary

Question

I recently presented my M.D work and one question got me into trouble.

Context:

In my neural network classification, I use 1 single output sigmoid neuron.

My problem requires that i classify the input into two groups : 0 or 1.

For the loss function, I use lasagne.objectives.binary_crossentropy :

Computes the binary cross-entropy between predictions and targets.

L= $−tlog(p)− (1−t) log(1−p)$

Question made in my presentation by a teacher

"You cant be using binary cross-entropy because entropy is just used when your data either has noises or some probability, why and how do you use that loss function? You cant use it because for each input the model already classify it to either 1 group, so 'entropy' makes no sense".

Im confused, especially by the last part, because my model has a sigmoid output that ranges from 0 to 1. I know that the output from the sigmoid neuron isnt exactly a probability, However his question doesnt make sense to me.

Would someone help me come up with an answer for that?

Sources:

http://lasagne.readthedocs.io/en/latest/modules/objectives.html#loss-functions

Answer 1

Your teacher needs to open a basic book on machine learning. The cross entropy loss function is equivalent to minimizing the KL divergence between your empirical and predicted distributions. While you can think of your predicted distribution as a probability, it's much more straightforward to think of it as a "confidence" in your prediction. If your teacher wants you to map your predicted distribution to 0s and 1s, then you would use ROC analysis to do so, since there is no universal way of mapping continuous values to discrete ones.

Answer 2

As you said in the comments, (at least in the training set) there are no conflicting patterns. Thus, it is a little unusual to talk about cross entropy when there is no confusion (entropy) in the data. A more natural choice could have been using a hinge loss (as SVMs do). On the other hand, Logistic Regression (LR) may also yield good results even if you do not expect a distribution on $P(y|X)$. In fact, results from LR may be useful if you need some sort of "confidence" on your classification. It is easier to interpret a probability than a distance to a hyperplane in a large dimensional space (in the case of the hinge loss). Or you may desire to expand your results beyond binary classification and LR can do this naturally. Without being familiar with your work it is impossible for us Internet strangers to guess for what reasons you chose cross entropy for your problem.

Aside from that (and perhaps more importantly), the qualification (and thesis/dissertation) committees also evaluate your mastery of the concepts used in your work. Thus, to obtain a Msc or Phd you need more than merely good results based on experimentation and/or trial and error. You need to understand your results and have an intuition or guiding principle on why you did what you did. In this setting there are many questions without a unique correct answer. It is more like a discussion than an exam. There are many possibilities on why this professor asked this question, but the important part is that you should be able to explain your design decisions and argue why you chose a certain method over other viable options.