Neural Network: For Binary Classification use 1 or 2 output neurons? Assume I want to do binary classification (something belongs to class A or class B). There are some possibilities to do this in the output layer of a neural network:


*

*Use 1 output node. Output 0 (<0.5) is considered class A and 1 (>=0.5) is considered class B (in case of sigmoid)

*Use 2 output nodes. The input belongs to the class of the node with the highest value/probability (argmax). 
Are there any papers written which (also) discuss this? What are specific keywords to search on?
This question is already asked before on this site e.g. see this link with no real answers. I need to make a choice (Master Thesis), so I want to get insight in the pro/cons/limitations of each solution. 
 A: Machine learning algorithms such as classifiers statistically model the input data, here, by determining the probabilities of the input belonging to different categories. For an arbitrary number of classes, normally a softmax layer is appended to the model so the outputs would have probabilistic properties by design:
$$\vec{y} = \text{softmax}(\vec{a}) \equiv \frac{1}{\sum_i{ e^{-a_i} }} \times [e^{-a_1}, e^{-a_2}, ...,e^{-a_n}] $$
$$ 0 \le y_i \le 1 \text{ for all i}$$
$$ y_1 + y_2 + ... + y_n = 1$$
Here, $a$ is the activation of the layer before the softmax layer. 
This is perfectly valid for two classes, however, one can also use one neuron (instead of two) given that its output satisfies:
$$ 0 \le y \le 1 \text{ for all inputs.}$$
This can be assured if a transformation (differentiable/smooth for backpropagation purposes) is applied which maps $a$ to $y$ such that the above condition is met. The sigmoid function meets our criteria. There is nothing special about it, other than a simple mathematical representation,
$$ \text{sigmoid}(a) \equiv \sigma(a) \equiv \frac{1}{1+e^{-a}}$$ 
useful mathematical properties (differentiation, being bounded between 0 and 1, etc.), computational efficiency, and having the right slope such that updating network's weights would have a small but measurable change in the output for optimization purposes.
Conclusion
I am not sure if @itdxer's reasoning that shows softmax and sigmoid are equivalent if valid, but he is right about choosing 1 neuron in contrast to 2 neurons for binary classifiers since fewer parameters and computation are needed. I have also been critized for using two neurons for a binary classifier since "it is superfluous". 
A: In the second case you are probably writing about softmax activation function. If that's true, than the sigmoid is just a special case of softmax function. That's easy to show.
$$
y = \frac{1}{1 + e ^ {-x}} = \frac{1}{1 + \frac{1}{e ^ x}} = \frac{1}{\frac{e ^ x + 1}{e ^ x}} = \frac{e ^ x}{1 + e ^ x} = \frac{e ^ x}{e ^ 0 + e ^ x}
$$
As you can see sigmoid is the same as softmax. You can think that you have two outputs, but one of them has all weights equal to zero and therefore its output will be always equal to zero.
So the better choice for the binary classification is to use one output unit with sigmoid instead of softmax with two output units, because it will update faster.
A: For binary classification, there are 2 outputs p0 and p1 which represent probabilities and 2 targets y0 and y1.
where p0, p1 = [0 1] and p0 + p1 = 1; y0,y1 = {0, 1} and y0 + y1 = 1.
e.g. p0 = 0.8, p1 = 0.2; y0 = 1, y1 = 0.
To satisfy the above conditions, the output layer must have sigmoid activations, and the loss function must be binary cross-entropy.
