How to verify extremely low error rates

Question

I am faced with trying to demonstrate through testing an extremely low error rate for a sensor (no more than 1 error in 1,000,000 attempts). We have limited time to conduct the experiment so we anticipate not being able to obtain more than about 4,000 attempts. I see no problem showing the sensor does not meet the requirement, as even one error in 4,000 attempts will yield a 95% confidence interval for the error rate with a lower limit greater than 0.000001. Showing that it does meet the requirement, however is the problem, as even 0 errors in 4,000 attempts still results in a lower bound greater than 0.000001. Any suggestions would be greatly appreciated.

Answer 1

This is a common problem, especially with modern components or systems which can have failure rates as low as $10^{-9}$. To address it, you need to make assumptions, create models, and/or incorporate other forms of data.

Lee Cadwallader of INL writes,

When no operating experience data exist for a component, such as a component in the design phase, the analyst has several options:

• Decomposition—deconstructing a component into its constituent parts and then assigning handbook failure rates to the parts. If the analyst is confident in the accuracy of part data, this technique is tedious but useful; if the data on parts are not accurate, other techniques should be used.

• Analyst judgment—may call for reverse estimation based on a system availability requirement or simply engineering judgment of the generic failure rates for that class of component.

• Expert opinion—obtaining qualitative opinions from subject matter experts and combining those to develop an order-of-magnitude failure rate.

• Component-specific techniques—for example, the Thomas method for piping.

Decomposition is frequently used for electronic parts, as evidenced by manuals of component failure rates.

Other sources suggest that industry data or experience can be used to inform, or in place of, testing data.

Other techniques discussed on Weibull.com include

In order to assess wear-out time of a component, long-term testing may be required. In some cases, a 100% duty cycle (running tires in a road wear simulator 24 hours a day) may provide useful lifetime testing in months. In other cases, actual product use may be 24 hours a day and there is no way to accelerate the duty cycle. High level physical stresses may need to be applied to shorten the test time. This is an emerging technique of reliability assessment termed QALT (Quantitative Accelerated Life Testing) that requires consideration of the physics and engineering of the materials being tested.

On a cautionary note, there appears to be a close parallel between this problem and that of estimating other rare events such as asteroid strikes and catastrophic failures in the financial system--Taleb's "black swans.". The latter rates were notoriously underestimated.

Answer 2

There's no way to prove an error rate < 1/1,000,000 with only 4,000 trials. You need to somehow select for errors (running more trials in parallel and only watching cases that result in an error) or apply some sort of stress that would increase the chance of an error, and then extrapolating from stressed conditions to normal conditions.

That's what geneticists would do, anyway....

Answer 3

Generally speaking, you can't. I would be very wary of techniques that claim to be able to prove a $1/10^6$ error rate given only $4000$ tests. Often those kinds of techniques involve somewhere making an assumption of independence, which there is no way to validate reliably: it's just a leap of faith. These kinds of flawed reasoning have led to serious failures in the world of safety-critical systems.

There may be some special cases where you can demonstrate the desired level of reliability using such a limited number of tests, e.g., by taking into account something about the physics of the situation. But they are rare, and that kind of reasoning is fragile.