# Linear Classifiers -- Single Linear Layer with k Neurons vs 'one vs rest' (k Linear Layers with 1 Neuron)

When using a linear classifier for a k-class classification problem, is there actually any difference of using

• a single linear layer with k neurons + cross entropy loss
• 'one vs rest': k linear layers with 1 neuron each + binary cross entropy loss [*]

In particular, should one be preferred over the other? Is there any crucial difference wrt convergence or convergence time? I apologize if this question seems very 'basic'.

[*] Detailed explanation: For this approach, k neurons are trained, one for each class and thus, the problem is reduced to k binary classification problems. For each of the neurons, either an observation belongs to the class the neuron is assigned to ('one') or not ('rest'). During inference time, the class of an observation is determined by feeding the observation into all k neurons and assigning the class of the neuron that produces the highest class score (simply corresponds to an argmax over the class scores).

• Can you describe your second option in more detail? Commented Oct 30, 2022 at 16:40
• @gunes just edited the question Commented Oct 30, 2022 at 16:49

In the first option, each neuron already follows a one vs. rest approach because its output is supposed to be closer to $$1$$ if only a sample from the corresponding class is given, and for other classes, it's expected to be closer to $$0$$. The decision is again based on the neuron with maximum output. So, assuming all neurons have the same output, e.g. sigmoid, architecture-wise, the two approaches are actually equivalent. It doesn't matter if they're trained separately or not because gradients do not clash.

But, their loss functions differ. In the first one, for a sample with class label $$u$$, the loss function is $$-\log p_u$$, where $$p_u$$ is the $$u$$-th neuron's output. In the second one, it is $$-\log p_u - \sum_{i\neq u}\log (1-p_i)$$. So, there is an extra term in the loss coming from the other neurons. You could have used the same loss function for the first architecture. So, it's not a matter of which arch. is useful or not, but a matter of loss function.

Unfortunately, there is no principled way to say which one is better. It depends on the problem, on the data, and the best way to find out is through validation. But, usually, another option is more common; and that is using a softmax at the end together with multi-class cross-entropy loss, which play along well in gradient-based opt. approaches.

• thanks! option 1 is actually intended to be the one you describe in the last paragraph (softmax + mutli-class cross-entropy loss); when phrasing the question, I unfortunately used the terminology of PyTorch, where it is just called "CrossEntropyLoss". So for clarification: is the approach of the last paragraph also called "one vs all"? Commented Nov 6, 2022 at 13:41