0
$\begingroup$

Why, in this case, is the test using a confidence interval not equivalent to the test using hypothesis testing, and is it?

Hypothesis testing for

$H_0: \pi = \pi_0$

$H_1: \pi\neq\pi_0$

A single trial has a Bernoulli distribution with mean $\pi$ and variance $\pi(1-\pi)$. According to the central limit theorem, the average of $n$ trials will have (approximately) mean $\pi$ and variance $\frac{\pi(1-\pi)}n$. Therefore,

$U_1 = \frac{\hat \pi -\pi}{\sqrt\frac{\pi(1-\pi)}{n}}$ is from $\mathcal N(0;1)$

If $H_0$ is true, $\pi = \pi_0$, so if $H_0$ is true,

$U_2 = \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ is from $\mathcal N(0;1)$.

So we reject $H_0$ iff ($u_\frac{\alpha}{2}> U_2$ or $u_{1-\frac{\alpha}{2}}< U_2$).

In other words, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$).

On the contrary, testing according to confidence intervals:

The confidence interval is

$\left[\hat\pi - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}};\hat\pi + u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}\right]$

We reject $H_0$ iff ($\pi_0< \hat\pi - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\pi_0> \hat\pi + u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$).

In other words, we reject $H_0$ iff ($\pi_0-\hat\pi < - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\pi_0-\hat {\pi} > u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$).

Rewriting the parenthesis:

($\hat\pi -\pi_0 > u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\hat {\pi} -\pi_0 < -u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$)

($\hat\pi -\pi_0 > u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$ or $\hat {\pi} -\pi_0 < u_{\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}$)

($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$)

The difference:

When using hypothesis testing, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$).

On the contrary, when using the confidence interval, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$).

Is the difference caused by replacing $\pi$ with $\hat\pi$ while constructing the confidence interval, making it just an approximate confidence interval? Or by something else?

(There are other questions on the site about the equivalence between confidence intervals and hypothesis testing, but none of them helped me with this.)

$\endgroup$
  • $\begingroup$ No they are the same tests (confidence interval is approximate of course because this is a large sample test). In fact the Wald CI is derived from this test itself (see math.stackexchange.com/questions/1448233/…). $\endgroup$ – StubbornAtom Jun 5 '19 at 7:04
  • $\begingroup$ @StubbornAtom They're not the same, because when we do hypothesis testing, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\pi_0(1-\pi_0)}{n}}$), but when we use the confidence interval, we reject $H_0$ iff ($u_\frac{\alpha}{2}> \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$ or $u_{1-\frac{\alpha}{2}}< \frac{\hat \pi -\pi_0}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}$). I guess that instead of asking if they're the same, I only should've asked what's the difference caused by. $\endgroup$ – Golden Gleam Jun 6 '19 at 9:05
1
$\begingroup$

In this case, if you use the value of the parameter under the null hypothesis in the standard error calculation in U2, then you are performing the score test. If you had instead produced a test statistic using the sample proportion instead in the standard error, you'd be performing the Wald test and you'd have consistency with rejecting based on the Wald confidence interval in your post. Both tests are valid asymptotic tests, but the score test has better properties in this case.

http://ocw.jhsph.edu/courses/MethodsInBiostatisticsII/PDFs/lecture18.pdf

$\endgroup$
  • $\begingroup$ Thanks, but I can derive the confidence interval that I used, simply from using the assumption that $U_1 = \frac{\hat \pi -\pi}{\sqrt\frac{\pi(1-\pi)}{n}}$ is from $\mathcal N(0;1)$. And yet I end up with a different rejection condition than in case of hypothesis testing. I suppose that what I'm really interested in is what's the difference caused by (since I'm starting with the same assumption in both cases). My guess was that it's caused by us replacing $\pi$ with $\hat\pi$ when deriving the confidence interval. Is that right? $\endgroup$ – Golden Gleam Jun 6 '19 at 9:12
  • $\begingroup$ @GoldenGleam You're not starting with the same assumption. The confidence interval is derived from the assumption that $\frac{\hat \pi -\pi}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}\sim N(0,1)$ This comes from central limit theorem and Slutsky. This is true for any value of $\pi$. The assumption that $U_2 \sim N(0,1)$ is only true if $\pi=\pi_0$, and is only based on CLT. $\endgroup$ – jsk Jun 6 '19 at 15:32
  • $\begingroup$ (Part 1) I derive the confidence interval from $U_1=\frac{\hat \pi -\pi}{\sqrt\frac{\pi(1-\pi)}{n}} \sim \mathcal N(0;1)$, therefore $P(u_\frac{\alpha}2 < \frac{\hat\pi - \pi}{\sqrt\frac{\pi(1-\pi)}n} < u_{1-\frac\alpha 2})=95\%$. Therefore, $P(\hat\pi-u_{1-\frac{\alpha}2}\sqrt\frac{\pi(1-\pi)}n<\pi<\hat\pi + u_{1-\frac{\alpha}2}\sqrt\frac{\pi(1-\pi}n)$. Afterwards I replace $\pi$ with $\hat\pi$ twice to obtain the confidence interval $\left[\hat\pi - u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}};\hat\pi + u_{1-\frac{\alpha}{2}}\sqrt{\frac{(1-\hat{\pi})\hat{\pi}}{n}}\right]$. $\endgroup$ – Golden Gleam Jun 7 '19 at 15:59
  • $\begingroup$ @GoldenGleam. You can't derive a confidence interval in that manner from $U_1$. You get stuck and then decide to replace because you got stuck is not a principled approach, and you are providing no justification regarding why you can make that substitution at that point. I gave you the technically correct way of constructing the interval, and it replaces it in step 1 using slutsky to demonstrate that after replacing, you still have $N(0,1)$. $\endgroup$ – jsk Jun 7 '19 at 16:06
  • $\begingroup$ (Part 2) Thanks, but that was just a part one. I thought it would be only an approximate confidence interval, since $\hat \pi$ is usually close to $\pi$. But now I understand that it's a real confidence interval, because $U = \frac{\hat \pi -\pi}{\sqrt\frac{\hat\pi(1-\hat\pi)}{n}}\sim\mathcal N(0,1)$ too. So the actual reason I got a different result is that my confidence interval belongs to a different random variable than the hypothesis testing (namely, $U$ instead of $U_1$). Is that right? $\endgroup$ – Golden Gleam Jun 7 '19 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.