Using the $Z$-test instead of the binomial test for small $n$ - effect on power and Type I error

Question

Setup

Let $M \sim \text{Bin}(p,n)$. Let us then state the following two hypotheses:

$$\begin{align} H_0: p = 0.5 \text{ vs } H_1: p > 0.5 \end{align}$$

To test these hypotheses, I could use the exact binomial test with the following decision function:

\begin{align*} \delta(M) = \begin{cases} 1, & \text{if } M > q_{(H_0,1-\alpha)}; \\ 0, & \mbox{otherwise.}\end{cases} \end{align*} Here, $q_{(H_0,1-\alpha)}$ denotes the ($1-\alpha$)-quantile of the ${\text{Bin}(0.5,n)}$-distribution.

Alternatively, I could transform the variable into a $Z$-statistic

$$Z_n = \frac{\sqrt{n}(\hat{p}-0.5)}{\sqrt{\hat{p}(1-\hat{p})}} \xrightarrow{d} \mathcal{N}(\sqrt{n}\frac{\hat{p}-0.5}{\sqrt{\hat{p}(1-\hat{p})}},1)$$

with $\hat{p} = M/n$ and corresponding decision function \begin{align*} \delta(Z_n) = \begin{cases} 1, & \text{if } Z_n > z_{(1-\alpha)}; \\ 0, & \mbox{otherwise.}\end{cases} \end{align*}

If $n$ is high enough, the two tests should have the same power and Type I errors. However, I am interested in how this transformation affects the Type I errors and the power of the my test if $n$ is low.

Problem description

I simulated 100'000 independent significance tests for each combination of $p_0 = 0.5$ and $p_1 \in [0.01,0.02,\dots,0.98,0.99]$ and calculated the average number of rejections (that is, $\delta(\cdot) = 1$ at $\alpha = 0.05$) for each pair of $p_0$ and $p_1$. The simulations were done by simulating independent draws from $\text{Bin}(p_1,n)$-distributions, followed by calculation of the two test statistics and comparison with the respective significance thresholds.

The results for $n=5$ and $n=10$ are shown below:

The dashed lines represent the theoretical power curves, the solid lines with circles represent the empirical ones (i.e. the ones I simulated). The two solid black lines indicate $1-\beta=0.05$ and $p_1=0.5$, respectively.

As expected, the Type I error of the binomial test is strictly below $\alpha = 0.05$ and the empirical curve nicely fits its theoretical counterpart. Also as expected (given that $n$ is too low for the normal approximation of $Zn$ to hold), the empirical power curve for $Z_n$ does not correspond to its theoretical counterpart. I did not expect, however, the considerable increase in power when using the $Z$-test instead of the Binomial test. Granted, for $n=5$, this increase in power is paid for by a considerable increase in Type I error. But already for $n=10$, the Type I error rate is almost completely controlled at the $\alpha$ threshold. Hence, my question:

Question

Given the results of my simulations and if I want to maximise the power of my statistical test while controlling for a pre-defined Type I error rate, can I use the $Z$-test for $n$ as small as $10$? And if yes, is there are any kind of theoretical argument backing up my simulations? I somehow feel like I've overlooked something obvious here, but can't seem to find out what...

Answer 1

It's important to pay close attention to what your attained significance levels are for a given rejection rule.

For example, at $n=10$, $p_0=0.05$, the reasonable rejection rules are

"reject if $X\geq 6$",
"reject if $X\geq 7$",
"reject if $X\geq 8$",
"reject if $X\geq 9$",
"reject if $X\geq 10$".

The attained significance levels if you do that are:

 6 37.7%  
 7 17.2%  
 8  5.47%  
 9  1.07%  
10  0.0977%  

These pertain whether you're doing the binomial test or you're using the normal approximation.

If you do it correctly, you must attain one of these rejection rates under $H_0$. There's no other possibility with a fixed rejection rule (you can attain levels in between if you use a randomized test but this doesn't pertain to your simulation).

So under the binomial situation, you appear to have selected the rejection rule "reject if $X\geq 9$". In the simulation, this gives very close to the above rate of about 1%.

Now the next higher attainable significance level occurs when the cutoff is 8 (note that the cutoff is in the rejection region). This gives a significance level of about 5.5%.

What are the attainable Z-scores in this $n=10,p_0=0.5$ situation?

 X   Z
 6 0.645
 7 1.380
 8 2.372
 9 4.216
10   Inf

(Your formula didn't include a continuity correction, so I haven't either)

You can't get any Z-values in between these! The distribution of your Z-statistic is discrete because the binomials you're generating can only have 0,1,2,...,10 successes. There are only 11 possible Z's of which we're looking at the five that are above 0.

The upper tail area of those z-values suggests you would have chosen the rejection rule "reject if $X\geq 8$", i.e. an actual 5.5% significance level. Your simulated rejection rate at the null seems a little lower than that but is reasonably consistent with it. (If it was anything but 5.47%, you didn't do what you said you did.)

There's no actual difference in these two tests other than the significance level you're conducting them at! They both end up applying what corresponds to a discrete rejection rule to a discrete test statistic.

If you're going to allow the Z approximation to choose a 5.5% significance level when you set $\alpha=0.05$ you should allow the binomial test that same degree of liberty or you're not comparing like with like. You're forcing the binomial to use a lower $\alpha$ (keeping $\alpha<0.05$) while allowing the Z-test to go higher than 5% and then holding it up as having better performance.

(If you do regard 5.5% significance level as "controlling type I error pretty nicely" why doesn't the binomial get to use it?)

A suitable comparison of power would choose the same significance level. This difference in significance level is what's driving the difference in power that you see in your results.