# “What is behind the door” problem when chance door will not open

EDIT: In response to suggestion, I'll include more real-world context.

I am trying to predict the presence of lead pipes in older houses for the purpose of renovations. I have N heterogeneous houses. I built a model to predict whether the house has lead pipes based on obvious factors like age, type, income of neighborhood, etc. Now I need to assess the accuracy of the model.

I have a base data set of historical records that crucially is based on in-person inspections where it is often the case that the inspecting persons can't get into the house to see whether there is lead or not (people don't trust, no one is home, etc). So the possible outcomes are:

[no one opens door / refuse to open,  door opens: no lead, door opens: lead present]


We can't know what's behind the unopened doors.

Second wrinkle; I use my classification model to send a different group of people to offer renovation services at high-probability of lead houses. These people verifiably have an easier time getting into the house to make a determination (people seem to like them more).

So, given that applying the classification model treatment makes it likelier that I receive more complete information about household lead status, how do I tell how good my model is vs. the no-model case? How can I disentangle the contributions of the model vs. just sending different people when evaluating "success: finding lead in houses"?

• Make a prediction for every door and compare what you observe to what you predicted. Obviously this maximizes the information you will get. You haven't supplied any indication that it would bias that information, either. These considerations suggest you can dispense with the doors and focus on how one measures a model that predicts any binary outcome. – whuber Sep 26 '16 at 17:55
• You will get better answers if you describe your real problem, with these details. – Kodiologist Sep 26 '16 at 17:59
• This sounds like an unbiased answer would require sending the "natural salespersons" to a random sample of houses not accessed on the first round, i.e. ignoring your model's predictions when choosing the houses to re-visit. – GeoMatt22 Sep 26 '16 at 19:37