Difference between using a confidence interval and using the z-score for a two-sided test?

Question

From what I have read so far, it seems that in a two-sided hypothesis test of the form:

$H_0: \mu = 10$

$H_A: \mu \ne 10$

where you choose a significance level of let's say $\alpha = 0.05$, there are two methods of conducting the hypothesis test.

1. Creating a 95% confidence interval around the sample mean, $\bar{x}$
2. Getting the z-score for $\bar{x}$ and doubling the corresponding p-value

Are these methods inherently the same thing? Or are there cases where depending on $\bar{x}$, we could reject $H_0$ in favor of $H_A$ for method #1 but not for method #2, etc?

Answer 1

Assume you have normal data with $\mu$ unknown and $\sigma$ known, so that a z test and z confidence interval are both appropriate.

In the z test, you reject $H_0: \mu = \mu_0$ vs $H_a: \mu \ne \mu_0$ at level $\alpha = 5\%$ precisely when $Z = \frac{\bar X - \mu_0}{\sigma/\sqrt{n}}$ has $|T| \ge 1.96.$ That is,

$$P(|Z| < 1.96) = P\left(-1.96 < \frac{\bar X - \mu_0}{\sigma/\sqrt{n}} < 1.96\right)\\ = P\left(\bar X-1.96\frac{\sigma}{\sqrt{n}} <\mu_0 < \bar X+1.96\frac{\sigma}{\sqrt{n}}\right) = 0.95$$

and a 95% (two-sided) confidence interval is $\bar X \pm 1.95\frac{\sigma}{\sqrt{n}}.$

Thus, the 95% CI that is based on putting half of the error probability in each tail can be comsidered an interval of 'non-rejectable' null values $\mu_0$ for the specified two-sided test.

Notes: (a) In the case discussed above, one says that the CI "inverts the test." Some confidence intervals are formed by inverting a test and some are not. So don't get the idea that this "duality" always holds true.

(b) Bear in mind that we say $\bar X \pm 1.95\frac{\sigma}{\sqrt{n}}$ is a 95% confidence interval for $\mu.$ Another possible (but seldom used) kind of 95% CI would put 2% error probability in one tail and 3% in the other tail, instead of 2.5% in each tail: $P(\bar X-2.054\sigma/\sqrt{n} \le\mu_0\le \bar X+1.881\sigma/\sqrt{n}) = 0.95.$ [CIs that put equal probability in each tail are sometimes called 'probability-symmetric'. Confidence intervals that are not probability-symmetric are mainly used for asymmetrical distributions.]

qnorm(c(.02, .97))
## -2.053749  1.880794
qnorm(c(.025, .975))
## -1.959964  1.959964