Computing p-value vs. constructing confidence interval from sample for proportions

Question

Let's say D is a Bernoulli distribution with parameter $\mu = 0.6$ and we want to test whether or not $\mu=0.5$. So,

Null hypothesis: $\mu = 0.5$

Alt hypothesis: $\mu \neq 0.5$.

Suppose we have a sample $X_1,\ldots, X_n \sim D$ for large $n$. Now, my understanding is there are two ways to do the hypothesis test. One approach is to assume the null hypothesis is correct and calculate the probability of seeing $\bar{X}_n$. If the null is correct, then we expect the distribution for the sample mean to be

$$N(0.5, (0.5*0.5)/n)$$

Then we can calculate the p-value. However, equivalently we can construct a confidence interval from our sample and reject the null hypothesis if 0.5 is not in our constructed range.

$$N(\hat{p}, \hat{p}(1-\hat{p})/n)$$

My confusion is, are these two methods really equivalent? Because the variance of the normal distribution would be smaller for the second approach, isn't it possible that you could reject the null hypothesis in the second approach, but fail to reject the null hypothesis in the first approach? Any intuitive explanation for why they are equivalent?

Answer 1

My confusion is, are these two methods really equivalent?

No the methods are indeed not equivalent.

Note that there are also many different ways to construct the confidence intervals (and different ways to express hypothesis tests). The use of the parameter estimate $\hat{p}$ in the expression $N(\hat{p}, \hat{p}(1-\hat{p})/n)$ is a simplification and does not give an exact interval. The justification is that $\hat{p}(1-\hat{p})$ and $p(1-p)$ do not differ much when the sample size is large enough.

See more on the Wikipedia page about different ways to construct confidence intervals for the binomial proportion.

• The interval based on $N(\hat{p}, \hat{p}(1-\hat{p})/n)$ corresponds to the Wald interval. (the hypothesis test that corresponds to this interval is the Wald test)
• The expression $N(0.5, (0.5*0.5)/n)$ is more related to the Wilson score interval.

See also Confidence interval / p-value duality: don't they use different distributions? in which the examples in the answer by Demetri Pananos and in some of the comments relate to the binomial proportion.

Answer 2

A confidence interval based on normal approximation for the Bernoulli where $\hat p(1−\hat p)$ (by the way $\hat p=\bar X_n$) is plugged in for the variance estimator involves two approximations (one by the Central Limit Theorem, the other by variance estimation) and is therefore not equivalent to a test that does not involve variance estimation.

There are confidence intervals for the Bernoulli that don't estimate the variance either though, see https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval. I believe that the Wilson score interval explained there will, by checking whether 0.5 is in it, give you a test equivalent to the one you discuss, i.e., with normal approximation but without estimating the variance (assuming that the test is two-sided).

Answer 3

Suppose you have $n = 100$ independent observations $X_i$ from a Bernoulli distribution with Success probability $p.$ Then $$T_{100} = \sum_{i=1}^{100} X_i \sim\mathsf{Binom}(n=100,p).$$

Suppose you want to test $H_0: p = 0.5$ against $H_a: p \ne 0.5$

In particular, you might observe $T = 38$ Successes in $n = 100$ trials. Then using binom.test in R, you get the following results:

binom.test(38, 100, .5)

        Exact binomial test

data:  38 and 100
number of successes = 38, number of trials = 100, 
 p-value = 0.02098
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.2847675 0.4825393
sample estimates:
 probability of success 
                   0.38 

The P-value of this test is $0.02098 < 0.05 = 5\%.$ so you reject $H_0$ in favor of $H_a$ at the 5% level of significance.

The P-value for this 2-sided test can be computed as $$P(T \le 38) + P(T \ge 62) = 2P(T \le 38) = 0.02097874,$$ where $T \sim \mathsf{Binom}(100, 0.5).$ Computation in R below.

2 * pbinom(38, 100, 0.5) 
[1] 0.02097874

The idea is to find the probability of a value as far or farther from the mean $np = 100(.5) = 50$ as is $38,$ in either direction.

If you want to use critical values, then you would reject if the observed total $T \le 39$ or $T \ge 61.$ Then the size of the test is $\alpha = 0.352.$ If you tried to use critical values $40$ and $60$ (instead of 39 and 51), then the size of the test would be $0.057,$ which exceeds 5%. Because of the discreteness of the binomial distribution, it is not possible to test at exactly the 5% level.

2*pbinom(39, 100, .5)
[1] 0.0352002
2*pbinom(40, 100, .5)
[1] 0.05688793

Also, the confidence interval $(0.285 0.483)$ for $p$ shown in the R output has very nearly the intended 95% coverage probability for all possible values of $p.$

In the figure below, the P-value of the test is the sum of the heights of the vertical black bars outside the vertical blue lines.

R code for figure:

t = 0:100;  PDF = dbinom(t, 100, .5)
hdr = "Null Dist'n BINOM(100, 0.5)"
plot(t, PDF, type="h", lwd = 3, main=hdr)
 abline(h=0, col="green2")
 abline(v=0, col="green2")
 abline(v = c(38.5, 61.5), col="blue")

---

Notes: (1) Because $n$ is sufficiently large that $T$ is approximately normal, then you could use an approximate normal test. With such a test you can pretend to test at the 5% level, but the standardized binomial statistic $Z = \frac{T - np_0}{\sqrt{np_0(1-p_0)}}$ does not take all of the values implied by $|Z| \ge 1.96.$

For $T = 38,$ the normal test statistic is $Z = -2.4$ and the P-value is about $0.016 < 0.05 = 5\%.$

t = 38;  n = 100;  p=0.5
z = (t - n*p)/sqrt(n*p*(1-p));  z
[1] -2.4
2 * pnorm(z)
[1] 0.01639507

(2) Various statistical programs do the exact test or the normal approximation, or both. Also, some give one or more style of confidence interval. Here is output from a recent release of Minitab software. The first output shown is essentially the same as for the exact test in R, but displayed differently:

Test and CI for One Proportion 

Test of p = 0.5 vs p ≠ 0.5

                                                   Exact
Sample   X    N  Sample p         95% CI         P-Value
1       38  100  0.380000  (0.284767, 0.482539)    0.021

Minitab's version of the approximate normal test is shown below; accordingly, it gives a different P-value $(0.16)$ than for the exact test, but still happens to reject at the 5% level. Of course, when you have two different tests it might happen that you reject with one and not with the other. [In particular, if you have $T = 38$ and $n=100.$ then Minitab's exact test will not reject at the 2% level and its approximate normal test will reject at that level.]

Test and CI for One Proportion 

Test of p = 0.5 vs p ≠ 0.5

Sample   X    N  Sample p         95% CI         Z-Value  P-Value
1       38  100  0.380000  (0.284866, 0.475134)    -2.40    0.016

(3) The two Minitab printouts give slightly different 95% CIs. @SextusEmpiricius provides a link to a Wikipedia page that shows several styles of CIs in common usage. It seems that the second Minitab printout gives the Wald 95% CI: With $\hat p = 38/100,$ it is $\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}},$ which computes to $(0.285,\, 0.475).$

p.hat = t/n
CI = p.hat + qnorm(c(.025,.975))*sqrt(p.hat*(1-p.hat)/n)
CI
[1] 0.284866 0.475134

The Wald interval is an asymptotic interval, which should be used only for large $n.$ It does not 'invert the test' and so should not be used as a substitute for the approximate normal test unless $n$ is very large (my personal rule is $n \ge 500).$

Various styles of CIs can give remarkably different results for some sample sizes and totals $T.$ Consequently, if you are going to use CIs to do tests, you might see frequent disagreement whether to accept or reject. In particular, some 'exact' 95% CIs are quite long in order to be sure to give at least 95% coverage for all possible values of $p.$ These intervals are less likely to reject $H_0$ at significance level 5%.