# How to train and assess prediction models when data are imperfect (miss some true cases)?

Consider the following example. Suppose your labeled dataset includes images of dogs and you want a computer program for recognizing dogs in images. The problem is that your data are imperfect because some of the dogs have been labeled but not all of them, say, the person labeling images wasn't very good at it and missed some (although it was good enough to ensure that all the labeled cases are correct).

Or, more formally, imagine that for every image the labeler has a probability $$Pr(\text{annotate dog on image}\mid\text{dogs on image}) = p$$
and $$Pr(\text{dogs on image} \mid \text{annotated dogs}) = q,$$ with $$p=1/3$$ and $$q=1$$.

How to evaluate the program? Common measures of precision and recall are somewhat "biased," given we only have a lower bound of the total relevant cases.

So, how to assess the precision and recall of the computer program?

• "the person labeling images wasn't very good at it and missed, say, 1 every 3 dogs". Let's say you have a 1/2 share of doggy images in your labeled dataset. Are the missing dogs populate the labeled dataset but were not labeled? – Alexey Burnakov Sep 9 '19 at 13:53
• @AlexeyBurnakov I edited the question trying to make clearer what the labeler does. Hope I addressed your comment. – mrb Sep 10 '19 at 0:17
• thanks, it is clear. I think, first of all, that precision is not influenced by this labelling inaccuracy. Denote it pre = true positives / found positives. The true positives are 100% true as you stated. Found positives are trained based on the true positives either. So you report how many trues in your model output. – Alexey Burnakov Sep 10 '19 at 8:57
• the different scenario is with recall, I want to think more on it. – Alexey Burnakov Sep 10 '19 at 8:57

I though once again about precision in your situation, and my comment has been misleading, I guess.

Consider this: we have an assumption that a model outputs 1 (found dog) randomly with a probability equal to an observed label share in training sample. This effectively means that the model is untrained, and the output probability is simply for calculation reasons.

With this setting you will get precision equal to the observed share of label = 1 in your sample, which follows from independence of the output and the label.

However, the true precision will be higher, because unlabel dogs will populate the samples with output equals 1 along with the labels equal 1. The true precision will be equal the share of true labels in your sample (which again follows form the independence assumption).

Now we go on, and this math becomes irrelevant if we think about the TRAINED model, which will break the assumtion of independence between outputs, observed labels, and (importantly) true labels because the model will try to learn what the dog is and tend to find them better than random one. With this in mind, I don't see how to answer your question exactly, and this is maybe only possible when you run many simulations involving trained model and you will have to know:

p(sample is true dog)


and estimate:

p(sample is true dog | output = 1)


Simulation on random data:

library(data.table)

set.seed(1)

dt <- data.table(
is_dog = rbinom(10000, 1, 0.5)
)

dt[
is_dog == 1
, is_label := rbinom(.N, 1, 1/3)
]

dt[
, is_label := ifelse(is.na(is_label), 0, is_label)
]

dt[
, is_output := rbinom(10000, 1, mean(dt[, is_label == 1]))
]

cat(
'observed precision = ,'
, mean(dt[is_output == 1, is_label == 1])
)

cat(
'true precision = ,'
, mean(dt[is_output == 1, is_dog == 1])
)


The first thing you can do is to go for a one-class classifier that is trained on the cases labeled as dog: this type of classifier will not get confused by the non-dog cases containing dogs, pointlessly trying to find differences between dogs labeled as such and dogs not labeled as such.

For the validation of one-class classifiers, it is crucial to select the test cases with care - otherwise you can end up with arbitrarily good specificity/negative predictive value which are however based on totally irrelevant non-dog cases (images that are extremely easy to decide non-dog). There's nothing wrong with testing them, but in addition you should test the most doglike of the non-dog cases.
As another example, consider drug authentication. Of course the classifier must be able to recognize a cheap, say, lactose-only tablet fake - but that's not sufficient. For authenticity, it also needs to reject a properly formulated tablet of the drug produced by another manufacturer.

Then, if reassessing the labels is possible, you'll want to reassess the positives found by your algorithm and probably also some cases that were considered barely non-dog as well as a (randomly selected) number of test cases that is sufficient to establish the proportion of false-negative dogs with a precision that is suitable for your application among the images not labeled as dog.

You can calculate or at least guesstimate beforehand the required number of cases you need in order to get the figures of merit with a suitable precision or to show that your classifier performs as well as specified.

If reassessing the labels is not possible, you can extend the concepts of many classifier performance measures such as precision, recall, accuracy etc. to situations with uncertain (probability that a case belongs to a class) or fuzzy (case belongs partially to a class) class labels. Uncertainty or fuzziness in the label can be expressed by using continuous labels ranging $$[0, 1]$$ for each class (dog in your case). What happens then to the figures of merit is that you get a range of possible values that are in agreement with the uncertainty of the label and the observed prediction.

We did that in a situation where the labeling was inherently uncertain or fuzzy:
C. Beleites, R. Salzer and V. Sergo: Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22.

In the dog case, you'd set the label for the no-dog images to $$Pr (dog | annotated~no~dog)$$ for class dog (you can in addition set it to $$Pr (no~dog | annotated~no~dog)$$ for class no dog). To calculate this probability, you'll need the (true) proportion of dog images in among the images and the proportion of no dog among all labels in addition to $$p$$.