Let me give an example. Assuming I have a classification technique for automatically measuring if a part in a car needs repair or not, it is my goal to check my classifier results against the ground truth of a test set. However, in my test set there are some items for which I cannot tell whether they need repair or not.
Obviously, I could exclude those items from my test set. However, I may introduce some bias by doing so. There are also some other unrelated constraints that prevent me from getting different test data. Is there a way to calculate a confidence interval or something similar? Ultimately, I would like to calculate accuracy, precision, and recall.