Confidence interval / p-value duality: don't they use different distributions? Idea:

*

*p-value is less than the level of significance if and only if the corresponding CI does not include the null value; and vica versa,

*the p-value is greater than the level of significance if and only if the corresponding CI does include the null value.

Is this idea always correct? Is it only correct for certain sampling distributions, like the normal, but not correct in general?
If the idea is correct in general, then why? The p-value is calculated using the distribution of the statistic conditional on H0, while the CI is calculated using the unconditional distribution of the statistic. These are two different distributions - how or why does this lead to the duality?
If the idea is not correct in general, could you provide counterexamples? Do people typically think that these counterexamples are a problem and somehow try to correct them, to make the idea still be true?
 A: This is not correct in general.  I'll provide a simple example.
A popular test for a binomial proportion is derived from the central limit theorem.  If $\pi$ is the true risk for the outcome, then asymptotically,
$$ p \stackrel{d}{\approx} \mathcal{N}[\pi, \pi(1-\pi) / n] $$
where $p$ is our estimated risk and $n$ is our sample size.  The test is then found by standardizing $p$ from this distribution using either the estimated risk in the variance (Wald test) or the risk under the null (Score test).  The test statistic is
$$ Z=\frac{p-\pi_{0}}{\sqrt{\frac{p\left(1-p\right)}{n}}} $$
and associated confidence intervals are
$$ \left(\widehat{\pi}_{L}, \widehat{\pi}_{U}\right)=p \pm Z_{1-\alpha / 2} \sqrt{p(1-p) / n}$$
Your points in the bullets are true for this test and the associated confidence interval because the latter is derived from the former.  However, they fail in general as many different confidence intervals exist for the binomial $^{1.}$ all with close to nominal coverage and slightly different widths.  It could be the case that the test of proportions as shown above yields a p-value small enough to reject the null, but a confidence interval other than the one I've posted covers the null value.
We can demonstrate this with some R code.  I'll calculate the confidence intervals for a range and outcomes using a Wilson score interval and the asymptotic interval.  You will see they do not line up exactly, meaning some intervals cover some values while others don't, even considering the same data are used to create both.  Hence using some intervals we would reject the null so to speak, while using others would lead to a failure to reject the null.
library(binom)


n = 20
x  = seq(0, n, 2)

a = binom.wilson(x, n)
b = binom.asymp(x, n)


plot(a$upper, b$upper, xlab = "Wilson Upper Limit", ylab='Asymptotic Upper Limit', type = 'l', col='red')
abline(0, 1)



References


*

*Brown, Lawrence D., T. Tony Cai, and Anirban DasGupta. Confidence intervals for a binomial proportion and asymptotic expansions. The Annals of Statistics 30.1 (2002): 160-201.

A: Basically the duality holds,
see also this question about the duality: Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?
I can think of two reasons to say that it doesn't hold (see below), but it is not because the duality is wrong and instead it is more about details and semantical (every kid has a parent, but that doesn't mean that each kid and each parent are pairs).
No, not correct 1
There is no single the p-value and single the confidence interval. Instead, there are multiple ways to define p-values and multiple ways to define confidence intervals.
So a particular confidence interval and particular construction of a p-value do not need to correspond with each other.
Yes, correct 1
But, there is a correspondence such that every confidence interval can be used as a hypothesis test, and confidence distributions could be used to compute p-values for particular parameters/hypotheses.
The reason is that a confidence interval contains the parameter p% of the time no matter what the true parameter is.
So given that a hypothesis is true, the probability that it is outside a p% confidence interval is p%. The false rejection probability, if you use confidence intervals, is p%.
The only cases where this does not work are when the confidence intervals are not exact. E.g. sometimes confidence intervals are approximations or estimates. But then, you should allow the same freedom for p-values which can also be approximations or estimates.
No, not correct 2
The other way around is not necessarily true. With every p-value (or more generally the construction method for a p-value) you can not always construct a confidence interval. Instead, sometimes you end up with a confidence region (a set of disjoint intervals).
A: Yes, both formulations are identical because this is just the definition of a confidence interval. Formally, if you have measured the parameter estimate $\theta_0$ for the parameter $\theta$, a $1-\alpha$ confidence interval $[\theta_1,\theta_2]$ is given by
\begin{eqnarray*}& & P(\hat{\theta} \geq \theta_0|\theta=\theta_1)=\alpha/2 \\
\mbox{and}\quad & & P(\hat{\theta}\leq \theta_0|\theta=\theta_2)=\alpha/2
\end{eqnarray*}
This definition distributes the error probability equally on both sides, so it is only equivalent to a two-sided test. Compared to the hypothesis testing scenario, the condition in the probability is the null hypothesis, which means that $[\theta_1,\theta_2]$ are just the borders of the rejection region.
Remark: Interestingly, most textbooks on statistics give a different defintion of a confidence interval: $P(\theta\in[\theta_1,\theta_2])=1-\alpha$. This definition depends on the unknown parameter value $\theta$ and only allows for solving for $\theta_{1/2}$ in very special cases: the asymptotic normal aproximation ("Wald interval") for the binomial proportion (which is problematic, as pointed out in other answers) and the confidence interval for a statsitical mean. I have found the general definition in

DiCiccio, Efron: "Bootstrap confidence intervals." Statistical Science, pp. 189-228, 1996

