I'm trying to multi-class classification problem with a small addition that I can't find a decent way to handle: according to domain knowledge, no classification is better than a wrong classification. More formally, if the set of classes in my training set is $\{a, b, c, d\}$, my classifier has to output one of $\{?, a, b, c, d\}$, where $?$ stands for "not classified", and the penalty for predicting $?$ instead of $a$ should be lower than predicting $b$ instead of $a$.

Obviously, this splits into two different parts, and the first is scoring. For now I ended up with a couple of manually written scorers, where "precision" is calculated as number of matched classes divided by number of "sure predictions" (predictions w/o $?$), and "recall" is calculated as number of matched classes divided by number of samples, and from there I calculate $F_\beta$ score, but I cannot help but think that there has to be more generic way of doing that, some way of expressing the fact that assigning $?$ class is better than incorrectly assigning any other class. In terms of confusion matrix, assigning $?$ is always a false positive (there's no $?$s in the domain), but it is a "better" false positive than any other.

(There's also an additional gimmick as I actually have an upper bound on my error rate, or, respectively, a lower bound on my "precision" as defined above, and I cannot find a way of integrating it into single metric, so I ended up adding two metrics to my cross validation/grid search in sklearn and simply monitor second one to be compliant)

And the second part would be training, but for some reason I expect this to be much easier part. For now I've extended existing sklearn classifier making it return None as prediction when some probability/confidence conditions are met and then I train its thresholds as a part of whole pipeline, but I think that I can actually add some dummy $x$x (all features zeroed out, for example) with an $y = ?$ to my training set and retrain the regular classifier to work with this. Would this work?

PS I tried to come up with a better title for this question, but I couldn't

  • $\begingroup$ This is interesting. It seems like an attempt to implement the "Correct answers receive +5 points, partially correct answers receive +1 point, incorrect answers receive -5 points" policy professors sometimes use for multiple-correct-answer multiple choice problems. $\endgroup$ – ERT Aug 9 '18 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.