4
$\begingroup$

I'm working with automating some data-retrieval, and now need to do some verification of the results. The system has automatically flagged 6% of the records, and I would like to determine a reasonable size of the remaining records to be manually verified, which will give me some assurance that other errors, if present, are identified.

Here's the quick numbers:

  • 2,502 total records
  • 152 (6%) known errors

Let's say I want an 80% chance of spotting errors in the remaining ~2,350 records. How many should I request be reviewed? Is there an easy formula to calculate it with different thresholds?

Update: Thanks for the input, everyone.

My goal is to determine a rough number of "good" records which should be manually reviewed to determine if there are still errors. These are all results which were labeled as "high confidence" by the automated system we used, so in theory we expect them to be 100% accurate.

I'm trying to be diligent in not fully trusting the automated process, but also conscientious of the cost of reviewing these. I would estimate that review will take approximately 1 minute per result; so reviewing all 2500 would be 40 hours of labor.

Improve this question
$\endgroup$
6
  • $\begingroup$ ...and as much as it sounds like it, this isn't actually homework. I'm importing geographic coordinates from addresses using a geocoding service; and need to find the balance of reasonable accuracy vs. labor cost $\endgroup$
    – STW
    Commented Feb 15, 2012 at 3:21
  • 1
    $\begingroup$ You might get a better answer at Cross Validated (stats.SE). $\endgroup$
    – Mike Spivey
    Commented Feb 15, 2012 at 4:38
  • $\begingroup$ Thanks for the input, I wasn't aware of the stats.SE site--I've flagged it and hope the admins will move it over for me :) $\endgroup$
    – STW
    Commented Feb 17, 2012 at 18:13
  • 2
    $\begingroup$ Are you asking about how trustworthy the automated procedure is? if so there isn't enough information i think. What you need is the number of correctly identified errors, the number of incorrectly identified errors, the number of correctly identified non-errors and the number of incorrectly idenified non-errors. I use the word "correct" in the sense of agreement with manual review. $\endgroup$
    – probabilityislogic
    Commented Feb 18, 2012 at 0:55
  • $\begingroup$ @STW, it is still not entirely clear what your objective is. I can try to guess, but... well... I am not a prophet. So: 1. Do you want the number $n$ so that you find at least one error with probability 80%? 2. Do you want $n$ so that there is an 80% probability that there are <i>no</i> errors in the remaining records? 3. If this is a cost optimization problem, you should cast is as such: write down your total cost (cost of automated check, manual check, and error), and figure out how to split the revision load between the computer and the human. $\endgroup$
    – StasK
    Commented Feb 18, 2012 at 5:30

1 Answer 1

Reset to default
1
$\begingroup$

Hypergeometric distribution to the rescue.

The complementary of finding at least one error in 2350 samples knowing that the occurence rate is 6% after 26 trials is 0.80. Checking just 48 makes you reach the 95% probability.

Some simple R code plotting the cumulative distribution follows:

> pe <- 0.06
> barplot(sapply(seq(55), function(i) 1-phyper(0,2350*pe, 2350*(1-pe), i)))

Of course that 6% does not apply anymore because if your system is any good the rate of occurrence should be lower on the remaining 2350 items but then I let you modify the variable pe accordingly.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.