5
$\begingroup$

EDIT: In response to suggestion, I'll include more real-world context.

I am trying to predict the presence of lead pipes in older houses for the purpose of renovations. I have N heterogeneous houses. I built a model to predict whether the house has lead pipes based on obvious factors like age, type, income of neighborhood, etc. Now I need to assess the accuracy of the model.

I have a base data set of historical records that crucially is based on in-person inspections where it is often the case that the inspecting persons can't get into the house to see whether there is lead or not (people don't trust, no one is home, etc). So the possible outcomes are:

[no one opens door / refuse to open,  door opens: no lead, door opens: lead present]

We can't know what's behind the unopened doors.

Second wrinkle; I use my classification model to send a different group of people to offer renovation services at high-probability of lead houses. These people verifiably have an easier time getting into the house to make a determination (people seem to like them more).

So, given that applying the classification model treatment makes it likelier that I receive more complete information about household lead status, how do I tell how good my model is vs. the no-model case? How can I disentangle the contributions of the model vs. just sending different people when evaluating "success: finding lead in houses"?

Improve this question
$\endgroup$
  • $\begingroup$ Make a prediction for every door and compare what you observe to what you predicted. Obviously this maximizes the information you will get. You haven't supplied any indication that it would bias that information, either. These considerations suggest you can dispense with the doors and focus on how one measures a model that predicts any binary outcome. $\endgroup$ – whuber♦ Sep 26 '16 at 17:55
  • $\begingroup$ You will get better answers if you describe your real problem, with these details. $\endgroup$ – Kodiologist Sep 26 '16 at 17:59
  • 1
    $\begingroup$ This sounds like an unbiased answer would require sending the "natural salespersons" to a random sample of houses not accessed on the first round, i.e. ignoring your model's predictions when choosing the houses to re-visit. $\endgroup$ – GeoMatt22 Sep 26 '16 at 19:37
1
$\begingroup$

It sounds like you want to check whether the houses you've been sending salesmen to are in fact particularly likely to have lead. Presuming the houses in the historical records weren't picked according to a prediction like this, and so they're more representative of all houses in the population, they're a good source of a base rate to compare your decisions to. Ignoring records for which an inspector wasn't allowed inside (which don't seem to be informative for our purposes), count the proportion of houses in the historical records that turned out to have lead. Then just compare that proportion A to the proportion B of houses you've sent salesmen to that had lead. The more that B is greater than A, the better you're doing at choosing where to send the salesman.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.