I have a multi-class short text classification task with a minor wrinkle: I'd like to also detect when the texts don't fit any of the classes well. I've tried to do it by simply adding unrelated texts into a separate class and learning an SVM, but with little success so far. Unsurprisingly, since 1) there are very many ways in which texts may not fit my classes, and 2) if I use too many examples for the "unrelated" class, the algorithm will simply learn to (nearly) always return it (and resampling reduces this to the previous problem).
I.e. this is a multi-class version of the problem solved by One-class SVMs. Are there standard solutions?
EDIT: I've come up with a possible solution (but not implemented or tested it yet).
Stage 1: a one-class classifier learned on the union of my classes (i.e. classify between relevant and irrelevant texts).
Stage 2: the usual multi-class classification if stage 1 says it's relevant.