Relation between power and sample size in a Binomial Test?

Question

$\beta$ (also called sensitivity or power) is a function of the sample size. It totally makes sense to me when we're performing a regression for example. The intuition is not that clear when we perform a binomial test. Consider the following example taken from this wikipedia page.

Suppose we have a board game that depends on the roll of a dice and attaches special importance to rolling a 6. In a particular game, the dice is rolled 235 times, and 6 comes up 51 times. If the die is fair, we would expect 6 to come up 235/6 = 39.17 times. Is the proportion of 6s significantly higher than would be expected by chance, on the null hypothesis of a fair die?

Let's assume the null hypothesis is wrong. Are we more likely to reject the null hypothesis if the sample size (here 235) is large?

Answer 1

I think $\beta$ is usually called type II error and $1-\beta$ is called power. If the sample size is bigger you will have a bigger power. Belowing are the R code to demonstrate the the sample size and power for the binomial distrbution. You will see bigger sample size has bigger power. The power for your case is 0.9445556 (two tailed) which I think pretty high.

power_n<-function(size){
p0<-51/235 #for calculate the critical region
c<-qbinom(p0,size, 1/6)
p<-seq(0.01,0.4,0.01)
pow<-1-dbinom(c,size,p)+dbinom(size-c,size,p) #the second part is very small can be omitted.
return(pow)
}
p<-seq(0.01,0.4,0.01)
plot(p,power_n(235),type="l",col="red")
lines(p,power_n(300),lty=2,col="blue")
lines(p,power_n(400),lty=3, col="black")

text(0.23,0.93,"Red, sample size =235")
text(0.23,0.94,"Blue, sample size =300")
text(0.24,0.95,"Black, sample size =400")

This is the power for OP's case (two tailed)

pow1=1-(dbinom(35,235,1/6)+dbinom(200,235,1/6))
pow1

Answer 2

Are we more likely to reject the null hypothesis if the sample size (here 235) is large?

Compared to when it's small? Yes, definitely.

You can directly compute the rejection probability under the assumptions at a given sample size as the population proportion moves up or down from $\frac16$ to other specific values of $p$ (and even if you couldn't calculate it, one could certainly simulate). Similarly, for some fixed effect size (e.g. $p=p_0+d$ for some $d$), you can compute the rejection probability.

So for example, you could compute the rejection probability at say $p=0.2$ for $n=59,235,$ and $940$ (the first and last being roughly a quarter and four times the $n$ in the original problem); you simply find the rejection rule giving your chosen significance level under the null, and then find the probability of being in that region under the alternative.

With large samples like those, you can calculate approximate rejection probabilities using normal approximations, but with software to compute the binomial distribution function, exact calculations can be done about as easily.