Does anyone know of a good reference that list the reasons for transforming multiple class classification problem into a set of binary sub-problems?

In response to comment: One reason to transform a multiple class classification problem into a set of binary sub-problems are that a binary network is usually simpler and therefore better suited to fit small data sets (Less prone to over fitting).

Furnkranz argues that a set of simpler binary classifiers can also provide better performance than one complex multiple class classifier because it is often easier to learn how to distinguish between two classes, than it is to learn how to distinguish between multiple classes. Simpler models reduce training time also.

Milgram shows that a set of binary SVM can preform better than a multi-class MLP. Though this might be due to choice of classifier rather than binarization.

If we focus on a MLP classifier, training a set of binary one-against-all MLPs and combining using softmax function would be very much equal the the multi-class MLP however this would allow us to use different hyper-paramater settings for each class.

  • $\begingroup$ It's going to be really interesting question if you expand it. Please describe your situation and what makes you to refrain from multi-class classification. $\endgroup$
    – ttnphns
    Feb 26, 2013 at 11:15
  • 1
    $\begingroup$ I think its often a question of statistical power and the relative occurrences of each class. In reality, most N-class classification problems will have some classes much better described (in terms of available data) than others due to a lower number of cases for some classes. Personally, I have always gotten better results when reducing a 3 class problem to a binary classification problem. $\endgroup$
    – BGreene
    Feb 26, 2013 at 16:12


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.