EDIT: In response to suggestion, I'll include more real-world context.
I am trying to predict the presence of lead pipes in older houses for the purpose of renovations. I have N heterogeneous houses. I built a model to predict whether the house has lead pipes based on obvious factors like age, type, income of neighborhood, etc. Now I need to assess the accuracy of the model.
I have a base data set of historical records that crucially is based on in-person inspections where it is often the case that the inspecting persons can't get into the house to see whether there is lead or not (people don't trust, no one is home, etc). So the possible outcomes are:
[no one opens door / refuse to open, door opens: no lead, door opens: lead present]
We can't know what's behind the unopened doors.
Second wrinkle; I use my classification model to send a different group of people to offer renovation services at high-probability of lead houses. These people verifiably have an easier time getting into the house to make a determination (people seem to like them more).
So, given that applying the classification model treatment makes it likelier that I receive more complete information about household lead status, how do I tell how good my model is vs. the no-model case? How can I disentangle the contributions of the model vs. just sending different people when evaluating "success: finding lead in houses"?