8
$\begingroup$

Idea:

  • p-value is less than the level of significance if and only if the corresponding CI does not include the null value; and vica versa,
  • the p-value is greater than the level of significance if and only if the corresponding CI does include the null value.

Is this idea always correct? Is it only correct for certain sampling distributions, like the normal, but not correct in general?

If the idea is correct in general, then why? The p-value is calculated using the distribution of the statistic conditional on H0, while the CI is calculated using the unconditional distribution of the statistic. These are two different distributions - how or why does this lead to the duality?

If the idea is not correct in general, could you provide counterexamples? Do people typically think that these counterexamples are a problem and somehow try to correct them, to make the idea still be true?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Differences typically arise from using asymptotic approximations. They do use different distributions, but if you do both test and CI "exactly", you should get correspondence (indeed some definitions of CIs are explicitly set up in terms of a correspondence). However, sometimes these "exact" procedures may be unsatisfying as tests or as CIs. $\endgroup$
    – Glen_b
    Jan 23 at 22:24

3 Answers 3

Reset to default
10
$\begingroup$

Basically the duality holds, see also this question about the duality: Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?

I can think of two reasons to say that it doesn't hold (see below), but it is not because the duality is wrong and instead it is more about details and semantical (every kid has a parent, but that doesn't mean that each kid and each parent are pairs).

No, not correct 1

There is no single the p-value and single the confidence interval. Instead, there are multiple ways to define p-values and multiple ways to define confidence intervals.

So a particular confidence interval and particular construction of a p-value do not need to correspond with each other.

Yes, correct 1

But, there is a correspondence such that every confidence interval can be used as a hypothesis test, and confidence distributions could be used to compute p-values for particular parameters/hypotheses.

The reason is that a confidence interval contains the parameter p% of the time no matter what the true parameter is.

So given that a hypothesis is true, the probability that it is outside a p% confidence interval is p%. The false rejection probability, if you use confidence intervals, is p%.

The only cases where this does not work are when the confidence intervals are not exact. E.g. sometimes confidence intervals are approximations or estimates. But then, you should allow the same freedom for p-values which can also be approximations or estimates.

No, not correct 2

The other way around is not necessarily true. With every p-value (or more generally the construction method for a p-value) you can not always construct a confidence interval. Instead, sometimes you end up with a confidence region (a set of disjoint intervals).

Improve this answer
$\endgroup$
3
  • $\begingroup$ I realize that there is no the CI. Though I thought that some CI's are better than others, under some criteria. Thus, under that criteria, there is or might be the best CI. But I thought that there is the p-value. Isn't the p-value = Pr(X < x | H0)? What other p-value can there be? If we know the distribution of X under H0, we can calculate the p-value, correct? $\endgroup$
    – Jessica
    Jan 24 at 2:30
  • $\begingroup$ Is the confidence distribution the same or similar to the distribution of the parameter that is not conditional on H0? This is fascinating. You're saying I can use this distribution to calculate the (a?) p-value? Is there a reference for this? $\endgroup$
    – Jessica
    Jan 24 at 2:55
  • 1
    $\begingroup$ @jessica take the example of Demetri Pananos, estimating the probability parameter of a binomial distribution. The p-value can be computed in just as many ways as the confidence interval can be computed. Instead of the wald statistic (which relates to the Wald interval) $$Z=\frac{p-\pi_{0}}{\sqrt{\frac{p\left(1-p\right)}{n}}}$$ we can also use the score (which relates to the Wilson score interval) $$Z=\frac{p-\pi_{0}}{\sqrt{\frac{\pi_0\left(1-\pi_0\right)}{n}}}$$ or use the exact distribution (which relates to the Pearson-Clopper interval). $\endgroup$
    – Sextus Empiricus
    Jan 24 at 6:30
6
$\begingroup$

This is not correct in general. I'll provide a simple example.

A popular test for a binomial proportion is derived from the central limit theorem. If $\pi$ is the true risk for the outcome, then asymptotically,

$$ p \stackrel{d}{\approx} \mathcal{N}[\pi, \pi(1-\pi) / n] $$

where $p$ is our estimated risk and $n$ is our sample size. The test is then found by standardizing $p$ from this distribution using either the estimated risk in the variance (Wald test) or the risk under the null (Score test). The test statistic is

$$ Z=\frac{p-\pi_{0}}{\sqrt{\frac{p\left(1-p\right)}{n}}} $$

and associated confidence intervals are

$$ \left(\widehat{\pi}_{L}, \widehat{\pi}_{U}\right)=p \pm Z_{1-\alpha / 2} \sqrt{p(1-p) / n}$$

Your points in the bullets are true for this test and the associated confidence interval because the latter is derived from the former. However, they fail in general as many different confidence intervals exist for the binomial $^{1.}$ all with close to nominal coverage and slightly different widths. It could be the case that the test of proportions as shown above yields a p-value small enough to reject the null, but a confidence interval other than the one I've posted covers the null value.

We can demonstrate this with some R code. I'll calculate the confidence intervals for a range and outcomes using a Wilson score interval and the asymptotic interval. You will see they do not line up exactly, meaning some intervals cover some values while others don't, even considering the same data are used to create both. Hence using some intervals we would reject the null so to speak, while using others would lead to a failure to reject the null.

library(binom)


n = 20
x  = seq(0, n, 2)

a = binom.wilson(x, n)
b = binom.asymp(x, n)


plot(a$upper, b$upper, xlab = "Wilson Upper Limit", ylab='Asymptotic Upper Limit', type = 'l', col='red')
abline(0, 1)

enter image description here

References

  1. Brown, Lawrence D., T. Tony Cai, and Anirban DasGupta. Confidence intervals for a binomial proportion and asymptotic expansions. The Annals of Statistics 30.1 (2002): 160-201.
Improve this answer
$\endgroup$
2
  • 5
    $\begingroup$ "Hence using some intervals we would reject the null so to speak, while using others would lead to a failure to reject the null." If some confidence intervals contain less/more often the null then this is not a reason to say that there is no duality with p-values. But instead, it is a reason to say that these confidence intervals are not accurate. $\endgroup$
    – Sextus Empiricus
    Jan 23 at 21:14
  • $\begingroup$ Thank you for your reply. I am not completely sure if this is a good example, because both CI's in your answer are approximations. Maybe if the exact CI is used, then the CI / p-value duality would hold. $\endgroup$
    – Jessica
    Jan 24 at 2:59
1
$\begingroup$

Yes, both formulations are identical because this is just the definition of a confidence interval. Formally, if you have measured the parameter estimate $\theta_0$ for the parameter $\theta$, a $1-\alpha$ confidence interval $[\theta_1,\theta_2]$ is given by \begin{eqnarray*}& & P(\hat{\theta} \geq \theta_0|\theta=\theta_1)=\alpha/2 \\ \mbox{and}\quad & & P(\hat{\theta}\leq \theta_0|\theta=\theta_2)=\alpha/2 \end{eqnarray*} This definition distributes the error probability equally on both sides, so it is only equivalent to a two-sided test. Compared to the hypothesis testing scenario, the condition in the probability is the null hypothesis, which means that $[\theta_1,\theta_2]$ are just the borders of the rejection region.

Remark: Interestingly, most textbooks on statistics give a different defintion of a confidence interval: $P(\theta\in[\theta_1,\theta_2])=1-\alpha$. This definition depends on the unknown parameter value $\theta$ and only allows for solving for $\theta_{1/2}$ in very special cases: the asymptotic normal aproximation ("Wald interval") for the binomial proportion (which is problematic, as pointed out in other answers) and the confidence interval for a statsitical mean. I have found the general definition in

DiCiccio, Efron: "Bootstrap confidence intervals." Statistical Science, pp. 189-228, 1996

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ The definition $P(\theta\in[\theta_1,\theta_2])=1-\alpha$ is used because it is more general. Your definition below is a very specific definition $$\begin{eqnarray*}& & P(\hat{\theta} \geq \theta_0|\theta=\theta_1)=\alpha/2 \\ \mbox{and}\quad & & P(\hat{\theta}\leq \theta_0|\theta=\theta_2)=\alpha/2 \end{eqnarray*}$$ For instance, we could just as well use $$\begin{eqnarray*}& & P(\hat{\theta} \geq \theta_0|\theta=\theta_1)=\alpha (1-x) \\ \mbox{and}\quad & & P(\hat{\theta}\leq \theta_0|\theta=\theta_2)=\alpha x \end{eqnarray*}$$ with any $x$ between $0$ and $1$. Your case is when $x=0.5$. $\endgroup$
    – Sextus Empiricus
    Jan 24 at 11:35
  • 2
    $\begingroup$ This definition depends on the unknown parameter value $\theta$ and only allows for solving for $\theta_{1/2}$ in very special cases This sounds like a critique that a different definition should be used. But it is the definition of a confidence interval. The definition is more general and allows multiple solutions for $\theta_{1/2}$ among which your definition is one of them. So if there is no easy solution for $\theta_{1/2}$ with this general definition then neither is there for the specific two-sided test based interval (otherwise that interval would be an easy solution). $\endgroup$
    – Sextus Empiricus
    Jan 24 at 11:41
  • $\begingroup$ Yes, distrubiting the error probability equally is just one option. The problem with the other definition is that it is one equation with three unkowns, and thus does not allow to solve for $\theta_1$ and $\theta_2$. $\endgroup$
    – cdalitz
    Jan 24 at 11:42
  • 1
    $\begingroup$ The unknowns are $\theta_1$ and $\theta_2$ just as in the two sided hypothesis test definition (in which case it also depends on $\theta$ and a simple solution may neither exist). Also, the definition of the confidence interval is not a formula for $\theta_{1/2}$ (because there is no unique interval that satisfies the definition of a confidence interval). $\endgroup$
    – Sextus Empiricus
    Jan 24 at 11:55
  • 1
    $\begingroup$ The definition of a confidence interval is not a formula to determine a confidence interval. This is because there is no unique confidence interval. The definition is just a necessary property of an interval to be a confidence interval. $\endgroup$
    – Sextus Empiricus
    Jan 24 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.