# Experimental Design on Testing Proportions

If you wanted to design an experiment to test the proportion of successes under two different strategies, and you had a fixed number of trials -- are you better off balancing the number of trials? (e.g. 20 and 20) -- or can it be advantageous to be unbalanced? (e.g. 10 and 40)

At first I thought you should balance them, but after thinking about it, I think it would depend on what the true proportions were. For instance, if p1=1, and p2=.5, you could probably pretty quickly be confident through testing that the first proportion is very high with fewer trials.

What equations would I need to reason through this more?

• I think you might want to check this: epitools.ausvet.com.au/content.php?page=CIProportion . Look at the +/- 95% CI values for the Jeffreys prior. One keyword of interest might be "acceptance sampling". Sep 19, 2016 at 14:11
• Or you could think about a sequential design. Mar 25, 2017 at 18:09
• Can you describe how is each "trial"? Is it a binomial experiment or simply bernoullu? at which $n$? Mar 25, 2017 at 18:10
• Based on a geometric distribution, select an ‘expected mean equalization rule’ design based the relative ratio of each underlying probability. So, for example, if an exploratory sequential sampling design or a very small fixed sample indicated sample probabilities of 0.1 and 0.5, respectively, the sample size for the p = 0.1 population would be five times larger. Knowing the total sample size n, you can now allocate samples between the populations. An alternate expected variance equalization rule yields, (0.5*(1-0.5))/(0.1*(1-0.1)), suggests 2.8 times, or use the mean of the two, 4 times. Nov 1, 2020 at 5:22

So you have two kind of Bernoulli experiments at disposition, corresponding to success probabilities $$p$$ and $$q$$. We will assume all trial runs are independent, so you will observe two random variables $$X \sim \text{Bin}(n,p) \\ Y \sim \text{Bin}(m,q)$$ and the total "budget" for observations is $$N$$, so $$m=N-n$$. Your question is, how should we distribute observations over $$n$$ and $$m=N-n$$? Is equal assignment best, that is $$n=m=N/2$$? or can we do better than that? Answer will of course depend on criteria of optimality.

Let us first do a simple analysis, which is mostly for hypothesis testing of the null hypothesis $$H_0\colon p=q$$. The variance-stabilizing transformation for the binomial distribution is $$\arcsin (\sqrt{X/n})$$ and using that we get that $$\DeclareMathOperator{\V}{\mathbb{V}} \V \arcsin(\sqrt{X/n}) \approx \frac1{4n} \\ \V \arcsin(\sqrt{Y/m}) \approx \frac1{4m}$$ The test statistic for testing the null hypothesis above is $$D = \arcsin(\sqrt{X/n}) - \arcsin(\sqrt{Y/m})$$ which, under our independence assumption, have variance $$\frac1{4n} + \frac1{4m}$$. This will be minimized for $$n=m$$, supporting equal assignment. Can we do a better analysis? There doesn't seem to be a (practical) way to invert this hypothesis test to get a confidence interval for $$p-q$$, so for that we do another analysis. We base such a confidence interval directly on $$X/n - Y/m$$, which have variance $$\frac{p(1-p)}{n}+ \frac{q(1-q)}{m}$$. Minimizing this in $$n$$ (with $$m=N-n$$) is an exercise in calculus, leading to the equation $$\frac{n}{N}=\frac1{\sqrt{Q}+1}$$ with $$Q=\frac{q(1-q)}{p(1-p)}$$. I will show this as a contour plot:

We can see from that plot that equal assignment is close to optimal for a large portion of the plot, specifically when the difference between $$p$$ and $$q$$ is not large. And, the proportion assigned to trial 1 is above 0.66 only in a very small part of the plot. So, unless you have reason to believe that one of the probabilities is very close to zero or one, and the other is far away from that, equal assignment will be close to optimal. Equal assignment will also be the "minimax" design, that it, it will minimize the maximum possible variance.

Below is the R code for the plot:

designBer  <-  Vectorize(   function(p, q)  {
Q  <-  q*(1-q)/(p*(1-p))
x  <-  1/(sqrt(Q)+1)  # ideal proportion in trial 1,  that is n/N
x}   )
p  <-  q  <-  seq(from=0.05,  to=0.95,  by=0.05)
pq  <-  expand.grid(p, q)
z  <-   outer(p, q, FUN=designBer)
image(p, q, z, main="Proportions of runs for trial 1")
contour(p, q, z, nlevels=19, xlab="p", ylab="q", add=TRUE)


Even more illuminating is maybe the plot of the relative efficiency (that is, ratio of variances) for using equal apportionment as compared to the optimal apportionment, again as a contour plot:

One can see that the relative efficiency is very close to 1 over a large part of the parameter space, and appreciably below that only if one of the probabilities is very close to zero or one, and the other far away. In that case detecting the difference will be easy, so loosing some efficiency doesn't really matter.

The R code for this second plot is below:

designRelEff  <-  Vectorize(  function(p, q)   {
Q  <-  q*(1-q)/(p*(1-p))
x  <-  1/(sqrt(Q)+1)        # ideal proportion in trial 1,  that is n/N
eff  <-  (2 * (p*(1-p) + q*(1-q) ) ) / ( ((p*(1-p))/x)  + (q*(1-q))/(1-x) )
1/eff
}
)
image(p, q, zz <- outer(p, q, FUN=designRelEff), main="Relative efficyency for equal apportionment" )
contour(p, q, zz,  nlevels=19, xlab="p", ylab="q", add=TRUE)


Finally, we give a bayesian variant for this experimental design. We assume the same binomial distribution for $$X$$ and $$Y$$ as above, with independent beta prior distribution for $$p$$ and $$q$$. We assume the same prior. This is a conjugated prior analysis, so the posterior distributions for $$p$$ and $$q$$ are (independent) beta distributions, with parameters $$\alpha+x,\beta+n-x$$ for $$p$$, for $$q$$ the parameters is $$\alpha+y,\beta+m-y$$. Using this we can calculate the posterior variance for $$\theta = p-q$$, which is the focus parameter for inference. That is a complicated expression I will not give here.

Now, one interesting idea for Bayesian design of experiments is to calculate the prior expectation of the posterior variance, this is called the preposterior expected variance. Again, this is a complicated expression I will not give, but a completely routine calculation.

Below I plot this preposterior variance function, for $$\alpha=\beta=0.5$$ (usual "noninformative" prior), as a function of $$n$$, for $$N=40$$:

We can see the minimum for $$n=20$$, but the function is rather flat around the minimum. The R code is given below:

makeprepost  <-  function(alfa=0.5,  beta=0.5) {
function(n, N=40) {
m  <-  N-n
(1/( (alfa+beta+n)^2 *(alfa+beta+n+1))) * ( alfa*(beta+n) + (beta+n-alfa)*(n*alfa)/(alfa+beta) -
n*alfa*beta/( (alfa+beta)^2) -
n*(n-1)*alfa*beta/( (alfa+beta)^2*(alfa+beta+1) ) -
(n*alfa/(alfa+beta))^2  )     +
(1/( (alfa+beta+m)^2 *(alfa+beta+m+1))) * ( alfa*(beta+m) + (beta+m-alfa)*(m*alfa)/(alfa+beta) -
m*alfa*beta/( (alfa+beta)^2) -
m*(m-1)*alfa*beta/( (alfa+beta)^2*(alfa+beta+1) ) -
(m*alfa/(alfa+beta))^2  )
}
}

prepost  <-  makeprepost(0.5,  0.5)
plot(prepost,  from=2,  to=38, xlab="n", col="red", ylab="preposterior expected posterior variance",
main="Preposterior analysis for p-q for N=40, alpha=beta=0.5")