# Detect bias in subset of Bernoulli processes

I'm looking for advice on the best method to use to answer this question.

General scenario:

We have multiple testing machines A,B,C,D etc. each tests a identical randomly selected part and provides a "Pass" or "Fail" result. All machines conduct the same test and can test any part but a machine may have defects that causes a bias towards passing/failing more then expected.

Question:

How do I detect testing machines that are biased (passing or failing more parts then expected) and quantify the degree of bias of their sample from the overall? Assuming that the number of biased machine results are small subsets of the overall number of results.

Thoughts:

Given the possible values are pass and fail this is a Bernoulli process? I was thinking that a chi-square goodness of fit test might be a way to approach this but I don't have a prior distribution to work with. The prior distribution would have to be measured from some initial testing phase using the test results? However then I'd be comparing test subsets against the prior generated from the combination of the subsets themselves, not good?

Another way to think of this would be N people flipping coins, some of which are biased. You would measure the results of each coins flips independent of the others and compare them to the expected result, if the results differ then you know the coins biased and don't need to know anything about the other coins.

My problem is that I do not know what the expected result should be and can only measure the results of the tests and compare each machines subset of results against the overall results?

• can you test the same parts on multiple machines? May 10 '13 at 1:27
• Hi Glen, Yes you can test any part on any machine. May 10 '13 at 9:39

There'd be several things you could do, but it sounds like perhaps a Generalized Linear Mixed Model might be in order.

The probability that a part is declared a failure would have a fixed effect (the machine) and a random effect (the part). You're not so much interested in the differences in the parts, only accounting for their unknown (common) effect, and the typical amount of variation they contribute. You are interested in identifying differences in the machines.

http://en.wikipedia.org/wiki/Generalized_linear_mixed_model

There are various packages that can fit these.

If I've understood the situation right, you wouldn't expect any kind of machine-part interaction, and if I remember how glmms work, you could do it this way in R, using the lme4 package:

testing.glmm1 <- lmer(TestResult ~ Machine + (1|Part), family="binomial")


where Machine and Part are factors (the first fixed, the second random), and TestResult is the 0/1 Fail/Pass response on each test on each machine. You'd then (firstly, at least) look for whether there's a Machine effect, since you're interested in detecting whether the alternative (that some machines differ in their underlying Pass/Fail rates) is the case.

If you haven't used GLMs it might pay to learn something about those first.