# How to estimate the minimum required sample size to reach a certain significance level for a binomial test?

I'm about to conduct an experiment where I will measure a binary variable. I will test two different settings of a machine and then I want check whether these two settings yield different proportions in the observed binary variable by using a binomial test.

Obtaining samples from that machine is expensive (both in terms of time and money) so I would like to take as few samples as possible to reach the following goals:

• If the two settings actually yield different proportions (this is what I expect) then I want to reach a significance level of $$p = 10^{-3}$$ with the binomial test.
• I accept a maximum 1 out of 1,000 chance (i.e. 0.1%) that this significance level will not be reached due to unlucky sampling, even though the two proportions are actually different.

That means with a probability of 99.9% I want to reach a significance level of $$10^{-3}$$ when drawing $$N$$ samples. The question is how large should $$N$$ be to fulfill these requirements?

I have a numerical model of that machine which gives me access to the theoretical proportions under both settings (e.g. $$p_1$$ and $$p_2$$). I suppose these need to be used in order to estimate the minimum sample size $$N$$?

• You want to have 99.9% power ("maximum 1 out of 1,000 chance (i.e. 0.1%) that this significance level will not be reached due to unlucky sampling"), to detect a difference between $p_1$ and $p_2$, with a type-I error rate of 0.001 ("a significance level of $p=10^{-3}$"): is that correct? What are the approximate values of $p_1$ and $p_2$ that are of interest; how big a difference between them do you care about?
– EdM
Commented Oct 6, 2020 at 15:46
• @EdM That's correct (I'm not very familiar with the terminology, but now after reviewing that, I can confirm that's exactly what I meant). The approximate values for the probabilities are $p_1\approx 0.50$ and $p_2\approx 0.99$, so there's quite a difference between them. Commented Oct 6, 2020 at 15:59

There are on-line tools for doing these calculations, for example here for your binomial two-sample test. Even with the stringent power (0.999) and Type-I error (0.001) requirements,* it's not hard to find a difference between $$p_1=0.5$$ and $$p_2=0.99$$ with a 2-sided test (which is what you should specify), 55 samples for each setting.
If instead you had wanted to distinguish $$p_1=0.50$$ from $$p_2=0.55$$ with the same specifications, you would have had a much harder time: 8113 samples each. It's easier to distinguish the same percentage-point difference near the ends of the binomial probability scale; to distinguish $$p_1=0.94$$ from $$p_2=0.99$$ you only need 1091 samples each. So use a proper power calculation instead of relying on gut instincts, as where you are along the probability scale and the difference you want to find matter a lot.