Given a classifier, that was trained on a data having two classes e.g. class red
and class blue
. The training samples are hand picked and only contains those two classes.
After training the classifier should classify some samples, but it turns out that those samples contain the classes red
, blue
and a third one magenta
(red+blue
) e.g a class that contains features of both classes. The classifier will correctly identify red
and blue
but will degrade to a random guesser for magenta
.
I try to give an example here (of course this is much simpler than in real life...):
assume that a sample has the features A
and B
, which are binary fields (either 0
or 1
).
The training data looks like this:
A B Label
-------------
1 0 red
1 0 red
0 1 blue
0 1 blue
One possibility for the classifier (e.g. a tree) would be to have the following output:
If A == 1: red
If A == 0: blue
Well, obviously this model is too general (but for this example I hope this makes no difference).
Now the classifier is tested on some samples:
A B Output
--------------
1 0 red
0 1 blue
1 1 red
Here is the problem.
The classifier will just output some "random" label for the third sample, depending on the training. If this oversimplified classifier would have looked at label B
, it would outputted blue
.
From other knowledge (e.g. a Gold-Standard) we can see that the third sample is actually neither red
or blue
but our magenta
class.
The classifier is just too general (?) or not well-trained (?). Is this a Sampling Error, because the third class was not identified as such during training (because the data was not available in the training set)?
How is this problem called exactly? I'm searching for the term but can not find one...
What could be a countermeasure? Of course, one could add samples of class magenta
into the training set. Are there other possibilities?
red
orblue
. The problem is, that the data I'm testing on has actually three classes, where the third (previously unknown) classmagenta
is exactly a mix of both classes the classifier was trained with. So if the classifier tries to classify such a sample, it will output eitherred
orblue
, depending where the threshold is set. $\endgroup$