70
$\begingroup$

Assume I want to do binary classification (something belongs to class A or class B). There are some possibilities to do this in the output layer of a neural network:

  • Use 1 output node. Output 0 (<0.5) is considered class A and 1 (>=0.5) is considered class B (in case of sigmoid)

  • Use 2 output nodes. The input belongs to the class of the node with the highest value/probability (argmax).

Are there any papers written which (also) discuss this? What are specific keywords to search on?

This question is already asked before on this site e.g. see this link with no real answers. I need to make a choice (Master Thesis), so I want to get insight in the pro/cons/limitations of each solution.

$\endgroup$
2
  • 1
    $\begingroup$ I think the OP of the linked question has a good point, the only difference is choice 2 has a larger number of parameters, is more flexible but more prone to over fitting. $\endgroup$
    – dontloo
    Apr 13, 2016 at 8:36
  • 2
    $\begingroup$ In the Udacity ML Nanodegree I learned that it's better to use one output node if the result is mutually exclusive simply because the network has less errors it can make. I think there are no pros in using 2 output nodes in that case but I have no scientific evidence for that $\endgroup$ Apr 11, 2017 at 17:33

3 Answers 3

44
$\begingroup$

In the second case you are probably writing about softmax activation function. If that's true, than the sigmoid is just a special case of softmax function. That's easy to show.

$$ y = \frac{1}{1 + e ^ {-x}} = \frac{1}{1 + \frac{1}{e ^ x}} = \frac{1}{\frac{e ^ x + 1}{e ^ x}} = \frac{e ^ x}{1 + e ^ x} = \frac{e ^ x}{e ^ 0 + e ^ x} $$

As you can see sigmoid is the same as softmax. You can think that you have two outputs, but one of them has all weights equal to zero and therefore its output will be always equal to zero.

So the better choice for the binary classification is to use one output unit with sigmoid instead of softmax with two output units, because it will update faster.

$\endgroup$
3
  • $\begingroup$ When you say one of them have all weights zero, do you mean the model didn't even consider one of the class during training? In practice, can we actually train this binary classifier with only one class of training data? $\endgroup$
    – deadcode
    Feb 12, 2019 at 12:36
  • 1
    $\begingroup$ It's more like threshold (bound) is fixed during the training and class. So that you know that if $x > 0$ than it's positive class and if $x < 0$ than it's negative class. With softmax you can learn different threshold and have different bound. RE weights with all zeros, I meant that sigmoid the same as softmax with 2 outputs for case when you have two output neutrons and one of the outputs $x$ and the other always $0$ no matter what was the input. It can be only when for the second output we have all weights equal to zero. I hope it helps. $\endgroup$
    – itdxer
    Feb 12, 2019 at 14:42
  • 1
    $\begingroup$ Note there are degenerate solutions of the form exp(x+alpha) / (exp(alpha) + exp(x+alpha)) - actually an infinite number of them - all yielding the same classification result as the one noted with weights all 0. The weights likely won't train to be all zero, but will train instead to be degenerate with the solution that has weights all 0. Avoid the (pointless and wasteful) degenerate solutions by using only one output neuron, it seems. $\endgroup$ Mar 2, 2019 at 4:50
7
$\begingroup$

Machine learning algorithms such as classifiers statistically model the input data, here, by determining the probabilities of the input belonging to different categories. For an arbitrary number of classes, normally a softmax layer is appended to the model so the outputs would have probabilistic properties by design:

$$\vec{y} = \text{softmax}(\vec{a}) \equiv \frac{1}{\sum_i{ e^{-a_i} }} \times [e^{-a_1}, e^{-a_2}, ...,e^{-a_n}] $$

$$ 0 \le y_i \le 1 \text{ for all i}$$ $$ y_1 + y_2 + ... + y_n = 1$$

Here, $a$ is the activation of the layer before the softmax layer.

This is perfectly valid for two classes, however, one can also use one neuron (instead of two) given that its output satisfies:

$$ 0 \le y \le 1 \text{ for all inputs.}$$ This can be assured if a transformation (differentiable/smooth for backpropagation purposes) is applied which maps $a$ to $y$ such that the above condition is met. The sigmoid function meets our criteria. There is nothing special about it, other than a simple mathematical representation,

$$ \text{sigmoid}(a) \equiv \sigma(a) \equiv \frac{1}{1+e^{-a}}$$

useful mathematical properties (differentiation, being bounded between 0 and 1, etc.), computational efficiency, and having the right slope such that updating network's weights would have a small but measurable change in the output for optimization purposes.

Conclusion

I am not sure if @itdxer's reasoning that shows softmax and sigmoid are equivalent if valid, but he is right about choosing 1 neuron in contrast to 2 neurons for binary classifiers since fewer parameters and computation are needed. I have also been critized for using two neurons for a binary classifier since "it is superfluous".

$\endgroup$
0
$\begingroup$

For binary classification, there are 2 outputs p0 and p1 which represent probabilities and 2 targets y0 and y1.

where p0, p1 = [0 1] and p0 + p1 = 1; y0,y1 = {0, 1} and y0 + y1 = 1.

e.g. p0 = 0.8, p1 = 0.2; y0 = 1, y1 = 0.

To satisfy the above conditions, the output layer must have sigmoid activations, and the loss function must be binary cross-entropy.

$\endgroup$
1
  • $\begingroup$ What does this add to the existing answers? $\endgroup$ Apr 7, 2022 at 21:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.