7
$\begingroup$

Suppose I am testing the effectiveness of a new piece of machinery. The current machine has a success rate of 20%, that is, for every hundred widgets produced, 20 can be sold.

Now I've developed a new prototype and I want to know if it performs better. Additionally, it only makes sense to invest in the new machine if it is 10% better, i.e. a 22% success rate.

So I set up an experiment and collect this data:

machine     attempts   successes
---------------------------------
Original      n1         k1       
Prototype     n2         k2

Assume that
* alpha= 5%
* beta= 20%
* I'd like to use the smallest sample size possible.

My understanding is that normally I'd want a 2-tailed difference of proportions test. But that test would only tell me if the two machines performed differently, not if one was 10% or more better.

How do I determine what sample size to use, what is the appropriate test(s) to use, and how do I report the magnitude of the difference?

Improve this question
$\endgroup$
6
  • $\begingroup$ How large are $n_1$ and $n_2$? Tests for comparing proportions tend to lose power if the sample sizes differ too much. $\endgroup$
    – MånsT
    Commented Jul 20, 2012 at 6:03
  • $\begingroup$ If the new machine has a $21.8\%$ success rate, you would like to reject it unless the original really only succeeds about $19.8\%$ of the time? Does testing the original cost the same as testing the prototype? $\endgroup$
    – Douglas Zare
    Commented Jul 20, 2012 at 6:16
  • $\begingroup$ @MånsT n1 and n2 could be between 1,000 to 100,000 if necessary. I can keep them roughly equal if that helps power. $\endgroup$
    – dan
    Commented Jul 20, 2012 at 13:46
  • $\begingroup$ @Douglas Zare testing both machines costs the same. The business case is that the cost of fully developing the prototype would only pay off if I can expect a 10% improvement. $\endgroup$
    – dan
    Commented Jul 20, 2012 at 14:05
  • $\begingroup$ @dan: I have some recommendations, but I don't have the time to post them now. I'll write an answer on Sunday or Monday unless threre already is a thorough answer by then! $\endgroup$
    – MånsT
    Commented Jul 20, 2012 at 20:17

1 Answer 1

Reset to default
0
$\begingroup$

You can do a one-sided test but even if you do a two-sided test the side to which you exceed the threshold clearly tells you if you are better or worse.

The standard hypothesis test only tells you that you are significantly better but not by the magnitude. To show that the magnitude is greater than a specified $\Delta$ requires a larger sample size. Instead of testing $p_1-p_2>0$ you test $p_1-p_2>\Delta$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.