1
$\begingroup$

I recently presented my M.D work and one question got me into trouble.

Context:

In my neural network classification, I use 1 single output sigmoid neuron.

My problem requires that i classify the input into two groups : 0 or 1.

For the loss function, I use lasagne.objectives.binary_crossentropy :

Computes the binary cross-entropy between predictions and targets.

L= $−tlog(p)− (1−t) log(1−p)$

Question made in my presentation by a teacher

"You cant be using binary cross-entropy because entropy is just used when your data either has noises or some probability, why and how do you use that loss function? You cant use it because for each input the model already classify it to either 1 group, so 'entropy' makes no sense".

Im confused, especially by the last part, because my model has a sigmoid output that ranges from 0 to 1. I know that the output from the sigmoid neuron isnt exactly a probability, However his question doesnt make sense to me.

Would someone help me come up with an answer for that?

Sources:

http://lasagne.readthedocs.io/en/latest/modules/objectives.html#loss-functions

$\endgroup$
  • $\begingroup$ Is your output perfectly represented by your input? If you have several different outputs for same input then you have some uncertainty (some call it noise, but I don't like that terminology) incorporated. In this case there is a distribution involved. $\endgroup$ – Cagdas Ozgenc Sep 28 '17 at 6:01
  • $\begingroup$ Its the same output for the same input. $\endgroup$ – KenobiShan Sep 28 '17 at 12:36
  • 1
    $\begingroup$ It seems like there was a miscommunication. I think you claimed that after the probabilistic output of your model you classified it as a zero or one given some threshold. However the log loss applies to your model before the threshold classification is applied, so it's calculated on the probabilistic outputs. I think your professor thought that you claimed that the $0$ or $1$ was the direct output of your model, to which cross-entropy cannot be applied. (If you wonder why this is, just substitute the values $0$ or $1$ in the cross-entropy expression and see what you get.) $\endgroup$ – Bridgeburners Sep 29 '17 at 12:46
3
$\begingroup$

Your teacher needs to open a basic book on machine learning. The cross entropy loss function is equivalent to minimizing the KL divergence between your empirical and predicted distributions. While you can think of your predicted distribution as a probability, it's much more straightforward to think of it as a "confidence" in your prediction. If your teacher wants you to map your predicted distribution to 0s and 1s, then you would use ROC analysis to do so, since there is no universal way of mapping continuous values to discrete ones.

$\endgroup$
1
$\begingroup$

As you said in the comments, (at least in the training set) there are no conflicting patterns. Thus, it is a little unusual to talk about cross entropy when there is no confusion (entropy) in the data. A more natural choice could have been using a hinge loss (as SVMs do). On the other hand, Logistic Regression (LR) may also yield good results even if you do not expect a distribution on $P(y|X)$. In fact, results from LR may be useful if you need some sort of "confidence" on your classification. It is easier to interpret a probability than a distance to a hyperplane in a large dimensional space (in the case of the hinge loss). Or you may desire to expand your results beyond binary classification and LR can do this naturally. Without being familiar with your work it is impossible for us Internet strangers to guess for what reasons you chose cross entropy for your problem.

Aside from that (and perhaps more importantly), the qualification (and thesis/dissertation) committees also evaluate your mastery of the concepts used in your work. Thus, to obtain a Msc or Phd you need more than merely good results based on experimentation and/or trial and error. You need to understand your results and have an intuition or guiding principle on why you did what you did. In this setting there are many questions without a unique correct answer. It is more like a discussion than an exam. There are many possibilities on why this professor asked this question, but the important part is that you should be able to explain your design decisions and argue why you chose a certain method over other viable options.

$\endgroup$
  • $\begingroup$ Thanks for the reply. However, the answer remains unclear. $\endgroup$ – KenobiShan Oct 1 '17 at 22:08
  • $\begingroup$ You mean the entropy, in this case, means the confidence? $\endgroup$ – KenobiShan Oct 1 '17 at 22:09
  • $\begingroup$ @KenobiShan Higher confidence means lower entropy. If this is not clear I suggest you to ask another question. $\endgroup$ – igordsm Oct 2 '17 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.