14
$\begingroup$

Several states in the USA are using the same statistical procedure for verifying that water service lines don't contain lead, and I can't figure out what it's based on because no references are cited.

Given a neighborhood of houses, this procedure (example from Colorado) dictates how many water service lines (connection from water supply pipe into house) need to be dug up and physically verified not to contain lead.

In order to be 95% confident with a 5% margin of error that there are no lead service lines in a water system, a random sample for physical verification must be conducted with the sample size determined by the [table below]. For example, after records review and no lead discovery, 5,000 service line materials remain unknown, 357 of those 5,000 service lines must be randomly selected and inspected; if all 357 of these service lines are non-lead, then the utility can be 95% confident there are no lead service lines in their distribution system.

You'd think they would use a Wilson Score Interval or the rule of three to calculate a 95% confidence interval for the fraction of lead service lines, but for the example above both methods give an upper confidence limit of 0.01, not the cited MOE of 0.05. I also don't understand why the sample size increases if the objective is to limit the fraction of lead service lines, not the total number of lead service lines.

enter image description here

lines   sample
1500    306
1600    310
1700    314
1800    317
1900    320
2000    322
2200    327
2400    331
2600    335
2800    338
3000    341
3500    346
4000    351
4500    354
5000    357
6000    361
7000    364
8000    367
9000    368
10000   370
15000   375
20000   377
30000   379
40000   381
60000   382
90000   383
$\endgroup$

2 Answers 2

13
$\begingroup$

This is a bit of a rabbit hole. It seems that the $357$ is the answer to a completely different question. Its calculation comes from the sample size for an infinite population of $$\dfrac{1.96^2 \times \tfrac12 \times \tfrac12}{0.05^2} \approx 384.16$$ and a finite population correction $$\dfrac{384.16}{1+\tfrac{384.16-1}{5000}}\approx 356.82$$

This represents the sample size required for a $95\%$ confidence interval which is $\pm 0.05$ from the sample proportion, when sampling without replacement from a population of $5000$.

It might be a reasonable approach to get a sample size producing a very rough estimate, for example if you wanted some idea of what proportion of lines had lead, and would be satisfied if seeing an answer of $50\%$ meant it should be interpreted as $45\%-55\%$.

It is completely inappropriate for trying to reach the conclusion that absolutely no lines have lead with some degree of confidence. If you really wanted an $x\%$ confidence figure for no lines having lead, you would need to test $x\%$ of the lines, so with $5000$ lines then testing $4750$ and seeing no lead would give $95\%$ confidence. This is intuitively sensible: if there was just one line with lead then the probability it would appear in your $x\%$ sample is $x\%$. Or use the hypergeometric distribution and R:

dhyper(0, 1, 5000-1, 4750)
# 0.05

Aiming for "absolutely no lines having lead" is a very high standard. Saying "$40$ or fewer lines out of $5000$ have lead" (i.e. $0.8\%$ or less) could be said with $95\%$ confidence on a sample size of $357$ and no positive tests: dhyper(0, 41, 5000-41, 357) gives 0.047. That is close to the "rule of three" for sampling without replacement, since $\frac3{357}(5000−357)\approx 39.02$.


This stems from the Flint lead-in-water crisis, leading to work done in the State of Michigan, later circulated nationwide in Annex E of an EPA report including the quoted numbers.

One of the links is to a white paper prepared for the Association of State Drinking Water Administrators (ASDWA) by BlueConduit. That then points to an online confidence-interval calculator which my browser objects to for having issues with its security certificate. You can then get it to reproduce the $357$ figure with a particular choice of numbers, as shown in this screen shot:

Sample size calculator

Another page on the same site says it gives the formula. Another screenshot:

Sample size formula

$\endgroup$
4
  • $\begingroup$ Thank you, very nice detective work. Do you agree that the Wilson Score Interval would be a tool better suited for this task? $\endgroup$
    – Jack Elsey
    Feb 7 at 23:16
  • 4
    $\begingroup$ @JackElsey For testing "absolutely no lines having lead" and a finite population sampled without replacement, I would use a null hypothesis of "at least one line has lead" and back-calculate from the hypergeometric distribution. I have added to my answer. $\endgroup$
    – Henry
    Feb 7 at 23:28
  • $\begingroup$ Thank you. Digging up 4750/5000 service lines would make quite a mess, though! It seems like the best option would be to get a risk assessor involved, so that potential health impacts are balanced with economic burden. $\endgroup$
    – Jack Elsey
    Feb 7 at 23:35
  • 2
    $\begingroup$ @JackElsey Aiming for "absolutely no lines having lead" is a very high standard. Saying "$40$ or fewer lines out of $5000$ have lead" (i.e. $0.8\%$ or less) could be said with $95\%$ confidence on a sample size of $357$ and no positive tests: dhyper(0, 41, 5000-41, 357) gives 0.047. That is close to the "rule of three" for sampling without replacement, since $\frac{3}{357}(5000-357)\approx 39.02$. $\endgroup$
    – Henry
    Feb 8 at 1:55
3
$\begingroup$

A couple of references I found:

The state of Colorado paper seems to parallel this one from Michigan

It references this group's work with this whitepaper

$\endgroup$
2
  • $\begingroup$ I don't think the ASTM standard and research article you link to are relevant (they are for testing water coming out of the pipe, not digging the pipe up), but the whitepaper definitely is. Unfortunately, that whitepaper doesn't provide any details about its methods except linking to a sample size calculator based on the normal approximation (which isn't valid for datasets with zero successes). I'm going to reach out to the authors and give them a chance to weigh in. Thank you for your answer. $\endgroup$
    – Jack Elsey
    Feb 7 at 23:03
  • 4
    $\begingroup$ @JackElsey The $357$ comes from the misuse of the sample size calculator $\endgroup$
    – Henry
    Feb 7 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.