0
$\begingroup$

I'm about to conduct an experiment where I will measure a binary variable. I will test two different settings of a machine and then I want check whether these two settings yield different proportions in the observed binary variable by using a binomial test.

Obtaining samples from that machine is expensive (both in terms of time and money) so I would like to take as few samples as possible to reach the following goals:

  • If the two settings actually yield different proportions (this is what I expect) then I want to reach a significance level of $p = 10^{-3}$ with the binomial test.
  • I accept a maximum 1 out of 1,000 chance (i.e. 0.1%) that this significance level will not be reached due to unlucky sampling, even though the two proportions are actually different.

That means with a probability of 99.9% I want to reach a significance level of $10^{-3}$ when drawing $N$ samples. The question is how large should $N$ be to fulfill these requirements?

I have a numerical model of that machine which gives me access to the theoretical proportions under both settings (e.g. $p_1$ and $p_2$). I suppose these need to be used in order to estimate the minimum sample size $N$?

Improve this question
$\endgroup$
2
  • $\begingroup$ You want to have 99.9% power ("maximum 1 out of 1,000 chance (i.e. 0.1%) that this significance level will not be reached due to unlucky sampling"), to detect a difference between $p_1$ and $p_2$, with a type-I error rate of 0.001 ("a significance level of $p=10^{-3}$"): is that correct? What are the approximate values of $p_1$ and $p_2$ that are of interest; how big a difference between them do you care about? $\endgroup$
    – EdM
    Commented Oct 6, 2020 at 15:46
  • $\begingroup$ @EdM That's correct (I'm not very familiar with the terminology, but now after reviewing that, I can confirm that's exactly what I meant). The approximate values for the probabilities are $p_1\approx 0.50$ and $p_2\approx 0.99$, so there's quite a difference between them. $\endgroup$
    – a_guest
    Commented Oct 6, 2020 at 15:59

1 Answer 1

Reset to default
3
$\begingroup$

This is a standard power analysis. To determine how many samples you need in each setting, you need to specify: a baseline value of interest, the difference between values that you want to detect, the chance that you are willing to accept a false-positive difference (Type-I error), and the chance that you are willing to accept that you miss a true-positive result (Type-II error). The power is 1 minus that Type-II error.

There are on-line tools for doing these calculations, for example here for your binomial two-sample test. Even with the stringent power (0.999) and Type-I error (0.001) requirements,* it's not hard to find a difference between $p_1=0.5$ and $p_2=0.99$ with a 2-sided test (which is what you should specify), 55 samples for each setting.

If instead you had wanted to distinguish $p_1=0.50$ from $p_2=0.55$ with the same specifications, you would have had a much harder time: 8113 samples each. It's easier to distinguish the same percentage-point difference near the ends of the binomial probability scale; to distinguish $p_1=0.94$ from $p_2=0.99$ you only need 1091 samples each. So use a proper power calculation instead of relying on gut instincts, as where you are along the probability scale and the difference you want to find matter a lot.

*Frequent choices are Type-I error of 0.05 and power of 0.8.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.