I would like to classify an input into one of 20 possible categories. I was wondering, what are the positives and drawbacks of using a neural network with 20 output layer neurons (each neuron represents the probability of the input belonging to a given class) compared to having 20 One VS All neural networks with 1 output neuron (probability that the input belongs to the given class) for each class?
My assumption is that having individual neural networks would result in greater accuracy since the former network would "lose complexity" as the number of output layer neurons increase. One obvious disadvantage of using the single neural network is a reduction in training time etc., but does this outweigh the potential increase in accuracy of using individual networks? Also, what if the number of classes increased to 1000?