0
$\begingroup$

Why, in this case, is the test using a confidence interval not equivalent to the test using hypothesis testing, and is it?

Hypothesis testing for

$H_0: \pi = \pi_0$

$H_1: \pi\neq\pi_0$

A single trial has a Bernoulli distribution with mean $\pi$ and variance $\pi(1-\pi)$. According to the central limit theorem, the average of $n$ trials will have (approximately) mean $\pi$ and variance $\frac{\pi(1-\pi)}n$. Therefore,

$U_1 = \frac{\hat \pi -\pi}{\sqrt\frac{\pi(1-\pi)}{n}}$ is from $\mathcal N(0;1)$

If $H_0$ is true, $\pi = \pi_0$, so if $H_0$ is true,

$U_2 = \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ is from $\mathcal N(0;1)$.

So we reject $H_0$ iff ($u_\frac{\alpha}{2}> U_2$ or $u_{1-\frac{\alpha}{2}}< U_2$).

In other words, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$).

On the contrary, testing according to confidence intervals:

The confidence interval is

$\left[\hat\pi - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}};\hat\pi + u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}\right]$

We reject $H_0$ iff ($\pi_0< \hat\pi - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\pi_0> \hat\pi + u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$).

In other words, we reject $H_0$ iff ($\pi_0-\hat\pi < - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\pi_0-\hat {\pi} > u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$).

Rewriting the parenthesis:

($\hat\pi -\pi_0 > u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\hat {\pi} -\pi_0 < -u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$)

($\hat\pi -\pi_0 > u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\hat {\pi} -\pi_0 < u_{\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$)

($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$)

The difference:

When using hypothesis testing, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$).

On the contrary, when using the confidence interval, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$).

Is the difference caused by replacing $\pi$ with $\hat\pi$ while constructing the confidence interval, making it just an approximate confidence interval? Or by something else?

(There are other questions on the site about the equivalence between confidence intervals and hypothesis testing, but none of them helped me with this.)

$\endgroup$
2
  • $\begingroup$ No they are the same tests (confidence interval is approximate of course because this is a large sample test). In fact the Wald CI is derived from this test itself (see math.stackexchange.com/questions/1448233/…). $\endgroup$ Jun 5, 2019 at 7:04
  • $\begingroup$ @StubbornAtom They're not the same, because when we do hypothesis testing, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$), but when we use the confidence interval, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$). I guess that instead of asking if they're the same, I only should've asked what's the difference caused by. $\endgroup$ Jun 6, 2019 at 9:05

1 Answer 1

1
$\begingroup$

In this case, if you use the value of the parameter under the null hypothesis in the standard error calculation in U2, then you are performing the score test. If you had instead produced a test statistic using the sample proportion instead in the standard error, you'd be performing the Wald test and you'd have consistency with rejecting based on the Wald confidence interval in your post. Both tests are valid asymptotic tests, but the score test has better properties in this case.

http://ocw.jhsph.edu/courses/MethodsInBiostatisticsII/PDFs/lecture18.pdf

$\endgroup$
13
  • $\begingroup$ Thanks, but I can derive the confidence interval that I used, simply from using the assumption that $U_1 = \frac{\hat \pi -\pi}{\sqrt\frac{\pi(1-\pi)}{n}}$ is from $\mathcal N(0;1)$. And yet I end up with a different rejection condition than in case of hypothesis testing. I suppose that what I'm really interested in is what's the difference caused by (since I'm starting with the same assumption in both cases). My guess was that it's caused by us replacing $\pi$ with $\hat\pi$ when deriving the confidence interval. Is that right? $\endgroup$ Jun 6, 2019 at 9:12
  • $\begingroup$ @GoldenGleam You're not starting with the same assumption. The confidence interval is derived from the assumption that $\frac{\hat \pi -\pi}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}\sim N(0,1)$ This comes from central limit theorem and Slutsky. This is true for any value of $\pi$. The assumption that $U_2 \sim N(0,1)$ is only true if $\pi=\pi_0$, and is only based on CLT. $\endgroup$
    – jsk
    Jun 6, 2019 at 15:32
  • $\begingroup$ (Part 1) I derive the confidence interval from $U_1=\frac{\hat \pi -\pi}{\sqrt\frac{\pi(1-\pi)}{n}} \sim \mathcal N(0;1)$, therefore $P(u_\frac{\alpha}2 < \frac{\hat\pi - \pi}{\sqrt\frac{\pi(1-\pi)}n} < u_{1-\frac\alpha 2})=95\%$. Therefore, $P(\hat\pi-u_{1-\frac{\alpha}2}\sqrt\frac{\pi(1-\pi)}n<\pi<\hat\pi + u_{1-\frac{\alpha}2}\sqrt\frac{\pi(1-\pi}n)$. Afterwards I replace $\pi$ with $\hat\pi$ twice to obtain the confidence interval $\left[\hat\pi - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}};\hat\pi + u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}\right]$. $\endgroup$ Jun 7, 2019 at 15:59
  • $\begingroup$ @GoldenGleam. You can't derive a confidence interval in that manner from $U_1$. You get stuck and then decide to replace because you got stuck is not a principled approach, and you are providing no justification regarding why you can make that substitution at that point. I gave you the technically correct way of constructing the interval, and it replaces it in step 1 using slutsky to demonstrate that after replacing, you still have $N(0,1)$. $\endgroup$
    – jsk
    Jun 7, 2019 at 16:06
  • $\begingroup$ (Part 2) Thanks, but that was just a part one. I thought it would be only an approximate confidence interval, since $\hat \pi$ is usually close to $\pi$. But now I understand that it's a real confidence interval, because $U = \frac{\hat \pi -\pi}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}\sim\mathcal N(0,1)$ too. So the actual reason I got a different result is that my confidence interval belongs to a different random variable than the hypothesis testing (namely, $U$ instead of $U_1$). Is that right? $\endgroup$ Jun 7, 2019 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.