11
$\begingroup$

I want to use deep learning to train a face/non-face binary detection, what loss should I use, I think it is SigmoidCrossEntropyLoss or Hinge-loss.

Is that right, but I also wonder should I use softmax but with only two classes?

$\endgroup$
9
$\begingroup$

Hinge loss and cross entropy are generally found having similar results. Here's another post comparing different loss functions What are the impacts of choosing different loss functions in classification to approximate 0-1 loss.

Is that right, but I also wonder should I use softmax but with only two classes?

Softmax is not a loss but a normalization function, it is often used together with the cross entropy loss, which is essentially equivalent to SigmoidCrossEntropyLoss. See also Cross-Entropy or Log Likelihood in Output layer

| cite | improve this answer | |
$\endgroup$
7
$\begingroup$

In general, when you have a problem where the sample can only belong to one class among a set of classes, you set the last layer to be a soft-max layer. It allows you to interpret the outputs as probabilities. When using a soft-max layer, cross entropy generally works very well, because the logarithmic term in the cross-entropy cancels out the plateau that is present in the soft-max function, therefore speeding up the learning process (think of points far away from $0$ in the sigmoid function).

In your case you have a binary classification task, therefore your output layer can be the standard sigmoid (where the output represents the probability of a test sample being a face). The loss you would use would be binary cross-entropy. With this setup you can imagine having a logistic regression at the last layer of your deep neural net.

Here are a couple links for you. Hope they help.
https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression
http://neuralnetworksanddeeplearning.com/chap3.html
https://www.quora.com/How-do-you-decide-which-loss-function-to-use-for-machine-learning

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Could you please shed some light on logistic regression at the last layer of a DNN? I have read the posts but can't really see the usefulness over sigmoid (as the last layer activation). Thanks $\endgroup$ – bit_scientist Jan 18 at 8:59
2
$\begingroup$

You could definitely use softmax with only 2 classes "Face" and "Not Face" and interpret the softmax output as confidence scores, which is a nice feature to get some intuition about your deep net.

Try both 2-class softmax and binary hinge loss. There's a recent paper Deep Learning using Linear Support Vector Machines using an SVM instead of a softmax classifier on top of deep conv nets and there's some promising results there.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Usually the logarithmic loss would be the preferred choice, used in combination with only a single output unit. Logarithmic loss is also called binary cross entropy because it is a special case of cross entropy working on only two classes.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You should update your first link. $\endgroup$ – nbro Jan 6 at 12:17
0
$\begingroup$

Theoretically, a softmax with 2 classes can be rewritten as a sigmoid, hence there should not be a difference in results between the two. Practically, as @dontloo mentioned, the number of parameters in the output layer would be double (not sure if this could lead to any overfitting problems), and of course you'd have 2 scores for the two classes (Face and Non_Face).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.