Suppose you have a classification problem in which you want to classify inputs into two exclusive classes (y1 and y2) with an artificial neural network (which models P(y|x)).
Among the two following architectures for the output layer, which one is better to model P(y|x)?
- use two output neurons, one for each class, with a softmax activation function. If a1 and a2 are the outputs of the two output neurons, P(y=y1|x)=a1 and P(y=y2|x)=a2 with a1+a2=1.
- use a single output neuron with a sigmoid activation function. If a is the output of the neuron, we can set P(y=y1|x)=a and P(y=y2|x)=1-a.
I can see the two following advantages, which suggests that the choice depends on the specific problem:
- In 1., the last layer has twice more parameters than in 2. and thus has more flexibility and can potentially model more complicated relationships.
- In 2., the last layer has twice less parameters than 1. and thus is less prone to overfitting.
