A colleague of mine has been developing a machine learning system with an active learning component. I was having trouble reproducing the metrics he's been reporting, until I found out that he's handling low-confidence results in an usual-to-me fashion. During cross-validation, he scores any value which is within the decision-boundary confidence threshold as a classification true positive, regardless of either labeled or predicted class. His justification is that the system "shouldn't be penalized" for results we want to have humans review.
This seems inappropriate to me, but I'm not very familiar with the active learning literature. Is this a common or otherwise well-supported practice?