Text classification for classes whose probabilities do not add to 1

Question

I have training data that classifies articles (article title and a summary) to one of two classes, let's say class A and class B

I want to be able to classify new articles. The problem is, the new articles that I want to classify can either be in class A or class B or they can not belong to either.

Should my training data include samples that do not belong to both classes? In other words, should I treat "none" as a separate class. Or is there another way (probably generative learning algorithms) where I can use class probabilities and makes decisions whether an article belong to a class based on the probability? I was using Multinomial Naive-Bayes but the results are very bad for articles that do not belong to either class

Answer 1

In the context of supervised learning, it's probably best to recode the data into 4 classes:

1. Neither class A nor class B
2. Class A only
3. Class B only
4. Both class A and class B

Note that you will indeed need training data for the new classes 1 and 4, or else the model can't learn anything about them.

Another option is to split this problem into two different classification problems: deciding whether or not a case is class A, and deciding whether or not a case is class B.

Answer 2

You might use outlier detection methods to find out other ones if most of your samples are actually from class A or B.

As you mentioned, another approach is to consider this problem as a classification problem with highly imbalanced classes. If you think you have representative examples for this other class, but they are just overwhelmed by A or B examples, you might try under/oversampling before running classifier.