6
$\begingroup$

Let's say D is a Bernoulli distribution with parameter $\mu = 0.6$ and we want to test whether or not $\mu=0.5$. So,

Null hypothesis: $\mu = 0.5$

Alt hypothesis: $\mu \neq 0.5$.

Suppose we have a sample $X_1,\ldots, X_n \sim D$ for large $n$. Now, my understanding is there are two ways to do the hypothesis test. One approach is to assume the null hypothesis is correct and calculate the probability of seeing $\bar{X}_n$. If the null is correct, then we expect the distribution for the sample mean to be

$$N(0.5, (0.5*0.5)/n)$$

Then we can calculate the p-value. However, equivalently we can construct a confidence interval from our sample and reject the null hypothesis if 0.5 is not in our constructed range.

$$N(\hat{p}, \hat{p}(1-\hat{p})/n)$$

My confusion is, are these two methods really equivalent? Because the variance of the normal distribution would be smaller for the second approach, isn't it possible that you could reject the null hypothesis in the second approach, but fail to reject the null hypothesis in the first approach? Any intuitive explanation for why they are equivalent?

$\endgroup$
7
  • 1
    $\begingroup$ Your sense of "equivalent" is a narrow one, because it focuses only on performance under the null (that is, the test size). An important aspect of performance is how the test does when the null is false: its power. Clearly these tests aren't anywhere near the same when it comes to the power to detect extreme departures from the null, because in those cases $\hat p(1-\hat p)$ is likely to be smaller than $(1/2)^2.$ So maybe you would find it more useful to study the entire risk function of each test and compare them. $\endgroup$
    – whuber
    Mar 6, 2022 at 20:34
  • 1
    $\begingroup$ It's not true that a test is based on only "calculating the probability of seeing $\bar X_n$". It is always based on the probability of "what was observed plus anything that is even less likely under the null hypothesis". This implies that there is a difference between one-sided and two-sided tests, and any approach to construct tests based on standard confidence intervals will correspond to two-sided tests, not one-sided ones (unless one uses confidence intervals ranging to (minus) infinity). $\endgroup$ Mar 6, 2022 at 20:41
  • $\begingroup$ A confidence interval based on normal approximation for the Bernoulli where $\hat p(1-\hat p)$ (by the way $\hat p=\bar X_n$) is plugged in for the variance estimator involves two approximations (one by the Central Limit Theorem, the other by variance estimation) and is therefore not equivalent to a test that does not involve variance estimation. (There are confidence intervals for the Bernoulli that don't estimate the variance either though.) $\endgroup$ Mar 6, 2022 at 20:44
  • $\begingroup$ Correction to my first comment: "confidence intervals ranging to (minus) infinity" should have said "ranging to the border of the parameter space", which here obviously is 0 or 1, not infinity. $\endgroup$ Mar 6, 2022 at 20:47
  • 1
    $\begingroup$ Does this answer your question? The equivalence of hypothesis testing and confidence intervals in the test of proportions $\endgroup$ Mar 7, 2022 at 17:36

3 Answers 3

6
$\begingroup$

My confusion is, are these two methods really equivalent?

No the methods are indeed not equivalent.

Note that there are also many different ways to construct the confidence intervals (and different ways to express hypothesis tests). The use of the parameter estimate $\hat{p}$ in the expression $N(\hat{p}, \hat{p}(1-\hat{p})/n)$ is a simplification and does not give an exact interval. The justification is that $\hat{p}(1-\hat{p})$ and $p(1-p)$ do not differ much when the sample size is large enough.

See more on the Wikipedia page about different ways to construct confidence intervals for the binomial proportion.

  • The interval based on $N(\hat{p}, \hat{p}(1-\hat{p})/n)$ corresponds to the Wald interval. (the hypothesis test that corresponds to this interval is the Wald test)
  • The expression $N(0.5, (0.5*0.5)/n)$ is more related to the Wilson score interval.

See also Confidence interval / p-value duality: don't they use different distributions? in which the examples in the answer by Demetri Pananos and in some of the comments relate to the binomial proportion.

$\endgroup$
3
  • $\begingroup$ Are there also different ways to construct test statistics? (Wald test, score test, likelihood ratio test, etc.) $\endgroup$
    – Alexis
    Mar 7, 2022 at 0:35
  • 2
    $\begingroup$ @Alexis, yes there are also different ways to construct the tests. The Wald test for the hypothesis test p=0.5 would not be using the variance based on p=0.5, but instead based on the observed estimate $\hat{p}$. $\endgroup$ Mar 7, 2022 at 0:39
  • $\begingroup$ So why the first sentence of your second paragraph? It seems to imply that every CI cannot be inverted into a corresponding test statistic. Is that true? $\endgroup$
    – Alexis
    Mar 7, 2022 at 16:53
3
$\begingroup$

A confidence interval based on normal approximation for the Bernoulli where $\hat p(1−\hat p)$ (by the way $\hat p=\bar X_n$) is plugged in for the variance estimator involves two approximations (one by the Central Limit Theorem, the other by variance estimation) and is therefore not equivalent to a test that does not involve variance estimation.

There are confidence intervals for the Bernoulli that don't estimate the variance either though, see https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval. I believe that the Wilson score interval explained there will, by checking whether 0.5 is in it, give you a test equivalent to the one you discuss, i.e., with normal approximation but without estimating the variance (assuming that the test is two-sided).

$\endgroup$
2
$\begingroup$

Suppose you have $n = 100$ independent observations $X_i$ from a Bernoulli distribution with Success probability $p.$ Then $$T_{100} = \sum_{i=1}^{100} X_i \sim\mathsf{Binom}(n=100,p).$$

Suppose you want to test $H_0: p = 0.5$ against $H_a: p \ne 0.5$

In particular, you might observe $T = 38$ Successes in $n = 100$ trials. Then using binom.test in R, you get the following results:

binom.test(38, 100, .5)

        Exact binomial test

data:  38 and 100
number of successes = 38, number of trials = 100, 
 p-value = 0.02098
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.2847675 0.4825393
sample estimates:
 probability of success 
                   0.38 

The P-value of this test is $0.02098 < 0.05 = 5\%.$ so you reject $H_0$ in favor of $H_a$ at the 5% level of significance.

The P-value for this 2-sided test can be computed as $$P(T \le 38) + P(T \ge 62) = 2P(T \le 38) = 0.02097874,$$ where $T \sim \mathsf{Binom}(100, 0.5).$ Computation in R below.

2 * pbinom(38, 100, 0.5) 
[1] 0.02097874

The idea is to find the probability of a value as far or farther from the mean $np = 100(.5) = 50$ as is $38,$ in either direction.

If you want to use critical values, then you would reject if the observed total $T \le 39$ or $T \ge 61.$ Then the size of the test is $\alpha = 0.352.$ If you tried to use critical values $40$ and $60$ (instead of 39 and 51), then the size of the test would be $0.057,$ which exceeds 5%. Because of the discreteness of the binomial distribution, it is not possible to test at exactly the 5% level.

2*pbinom(39, 100, .5)
[1] 0.0352002
2*pbinom(40, 100, .5)
[1] 0.05688793

Also, the confidence interval $(0.285 0.483)$ for $p$ shown in the R output has very nearly the intended 95% coverage probability for all possible values of $p.$

In the figure below, the P-value of the test is the sum of the heights of the vertical black bars outside the vertical blue lines.

enter image description here

R code for figure:

t = 0:100;  PDF = dbinom(t, 100, .5)
hdr = "Null Dist'n BINOM(100, 0.5)"
plot(t, PDF, type="h", lwd = 3, main=hdr)
 abline(h=0, col="green2")
 abline(v=0, col="green2")
 abline(v = c(38.5, 61.5), col="blue")

Notes: (1) Because $n$ is sufficiently large that $T$ is approximately normal, then you could use an approximate normal test. With such a test you can pretend to test at the 5% level, but the standardized binomial statistic $Z = \frac{T - np_0}{\sqrt{np_0(1-p_0)}}$ does not take all of the values implied by $|Z| \ge 1.96.$

For $T = 38,$ the normal test statistic is $Z = -2.4$ and the P-value is about $0.016 < 0.05 = 5\%.$

t = 38;  n = 100;  p=0.5
z = (t - n*p)/sqrt(n*p*(1-p));  z
[1] -2.4
2 * pnorm(z)
[1] 0.01639507

(2) Various statistical programs do the exact test or the normal approximation, or both. Also, some give one or more style of confidence interval. Here is output from a recent release of Minitab software. The first output shown is essentially the same as for the exact test in R, but displayed differently:

Test and CI for One Proportion 

Test of p = 0.5 vs p ≠ 0.5

                                                   Exact
Sample   X    N  Sample p         95% CI         P-Value
1       38  100  0.380000  (0.284767, 0.482539)    0.021

Minitab's version of the approximate normal test is shown below; accordingly, it gives a different P-value $(0.16)$ than for the exact test, but still happens to reject at the 5% level. Of course, when you have two different tests it might happen that you reject with one and not with the other. [In particular, if you have $T = 38$ and $n=100.$ then Minitab's exact test will not reject at the 2% level and its approximate normal test will reject at that level.]

Test and CI for One Proportion 

Test of p = 0.5 vs p ≠ 0.5

Sample   X    N  Sample p         95% CI         Z-Value  P-Value
1       38  100  0.380000  (0.284866, 0.475134)    -2.40    0.016

(3) The two Minitab printouts give slightly different 95% CIs. @SextusEmpiricius provides a link to a Wikipedia page that shows several styles of CIs in common usage. It seems that the second Minitab printout gives the Wald 95% CI: With $\hat p = 38/100,$ it is $\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}},$ which computes to $(0.285,\, 0.475).$

p.hat = t/n
CI = p.hat + qnorm(c(.025,.975))*sqrt(p.hat*(1-p.hat)/n)
CI
[1] 0.284866 0.475134

The Wald interval is an asymptotic interval, which should be used only for large $n.$ It does not 'invert the test' and so should not be used as a substitute for the approximate normal test unless $n$ is very large (my personal rule is $n \ge 500).$

Various styles of CIs can give remarkably different results for some sample sizes and totals $T.$ Consequently, if you are going to use CIs to do tests, you might see frequent disagreement whether to accept or reject. In particular, some 'exact' 95% CIs are quite long in order to be sure to give at least 95% coverage for all possible values of $p.$ These intervals are less likely to reject $H_0$ at significance level 5%.

$\endgroup$
3
  • 1
    $\begingroup$ The question is about comparing two tests, so I was disappointed not to find any analysis of the second test here. Did I miss something? $\endgroup$
    – whuber
    Mar 7, 2022 at 1:14
  • $\begingroup$ @whuber. My impression was that the first answer had already discussed normal tests, so I mentioned the possibility only briefly. May add to this later. $\endgroup$
    – BruceET
    Mar 7, 2022 at 1:21
  • $\begingroup$ My sense is the question concerns what test statistic to use "for large $n$:" the one based on a sample estimate of $\hat p$ for the Normal approximation or the one based on the specified value of $p$ under the null hypothesis. Ultimately this determination is made by comparing the risk curves (that is, actual confidence plus the entire power curve). $\endgroup$
    – whuber
    Mar 7, 2022 at 2:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.