improving classification using examples from 'other' classes I'm trying to classify a a small dataset of around 150 patients each with one of four different diseases. For each patient I have around 70 different features.
I also have an "extra" dataset of patients with 'other' diseases (not belonging to the 4 I need to classify). These are all different diseases that the classifier is not supposed to work on. I was wondering if I can use this extra dataset to somehow regularize the classifier for those four specific diseases that I do care about. So far these are the ideas I came up with:
1) Do softmax regression and input the extra dataset with labels [0.25,0.25,0.25,0.25]. This way I'm telling the classifier that these extra patients have labels that are "equally different" from the labels it needs to classify.
2) Use logistic regression and input the extra dataset with labels [0,0,0,0]. This way I'm telling the classifier the the extra patients have labels that are not of any of the labels that it should classify.
I'd like to hear other methods/opinions about the methods above/any other tips you might think are relevant to the problem.
 A: You could simply include those other diseases as a fifth class (or a couple more different classes depending on how homogeneous those records are and which classifier you use). You could also add another class for healthy people. As long as you have the same kind of input data for all those records, that's feasible.
Now, is it a good idea to do so? It depends:


*

*If eventual other classes are have records that are very different in the input feature space than any of your first 4 classes, you don't accomplish anything by adding more classes that don't interest you. You will just add computational complexity (more records, more classes, bigger models...)

*If some of the other diseases can look similar (as per your input features) to your 4 primary classes and you are certain that all your labels from the classes 1-4 but also 5+ are correct, then you should add the new class(es). This will prevent you from falsely labeling some test records as one of the first 4 classes because the additional classes will set stricter and more realistic decision boundaries around the initial classes.

*If the new classes can resemble the old ones in the input feature space but you are not certain of your labels, then all bets are off. It's unsure whether you will be preventing false labeling by adding more classes or encouraging it or both and if both which effect is stronger (it might still be worth it if the former effect is larger but that's very difficult to assess).


To know whether the new classes are close to the old ones, you can just add them to your model. Split your data in a training- and a test set and you will quickly have an idea if the classes 5+ are easy or hard to distinguish from the first 4. This experiment can however not tell you if you have a label certainty problem. For that, you need domain knowledge of how the labels in your data-set were derived and how sure you can be of them.
The only reasons to not just treat those new classes as any other classes (i.e. to search for a way to tell your classifier mathematically that you are less- or at least differently interested in them) are related to misclassification costs. If it is more costly to classify one of the initial four diseases as one of the additional ones than to to the opposite, perhaps because those four are more serious diseases that require more ,immediate treatment, then you need to take precautions. Just look up the many answered questions on this site regarding unbalanced data-sets. If that is not the case, treat eventual additional classes just as any other.
