Should I develop a binary classifier or a multi-class classifier with my data?

Question

I have a labeled set of data which contains 10 classes and ~400 training examples for each class. I would like to develop a classifier using this data.

However, out of the 10 classes, I am only interested if something is class 1 or not class 1. Hence, I am unsure if I should simply create a binary classifier as opposed to creating a multi-class classifier.

I am also aware that each class should have roughly the same amount of training data as to not develop a skewed classifier.

Thus, if I want to develop a binary classifier I will need to use two classes: class 1 and not class 1, where each class has ~400 training examples. This will result in only using ~45 training examples from each of the nine classes that are not class 1 to develop the new not class 1 class.

The problem is I am not sure if the act of using less data in my binary classifier will result in it being a worse classifier than if I created a multi-class classifier using all of my data.

Answer 1

The question is sort of subjective and I am answering based on personal experience. Given that you requirement is binary classification, I'd suggest you go with a binary classifier. Corresponding evaluation of this classifier will make sense too.

Yes, skew in class ratio can be bad, but if your model encounters this skew during prediction, then it should also be trained in the same environment. So I'd use all the training data with labels 1 and ~1.

Hope this helps.