Multiple output neurons or One VS All

Question

I would like to classify an input into one of 20 possible categories. I was wondering, what are the positives and drawbacks of using a neural network with 20 output layer neurons (each neuron represents the probability of the input belonging to a given class) compared to having 20 One VS All neural networks with 1 output neuron (probability that the input belongs to the given class) for each class?

My assumption is that having individual neural networks would result in greater accuracy since the former network would "lose complexity" as the number of output layer neurons increase. One obvious disadvantage of using the single neural network is a reduction in training time etc., but does this outweigh the potential increase in accuracy of using individual networks? Also, what if the number of classes increased to 1000?

Answer 1

Lets assume you went ahead with training the 20 individual neural networks. The next problem you would face is how to combine the 20 outputs so that you can get a class probability output showing class with highest probability, next highest, and so on. If you do this after training the models, and you have a tie between 2 or more models, then you would need to find a way to give precedence to one over the other.

All of this is easily avoided if you create a single network with 20 outputs, typically with a softmax activation function. This ensures that class conflicts are penalised during training and automatically taken care of.

As for your concerns on the training accuracy, you could address these by experimenting with adding more layers or more neurons in the last but one layer, or increase regularisation.