2
$\begingroup$

Let's say I make the claim that 9 out of 10 dentists recommend a tooth paste. Next, I can ask 1000 dentists, and let's say that I find 700 / 1000 acually recommend it.

so here $p_o = 0.9$, and $p = 0.7$

Now I want to do a z test on that to see if the initial claim is true based on the people I asked.

I've seen examples where the standard error for that is computed as

$SE = \sqrt{\frac{p_o(1 - p_o)}{n}}$

I'm wondering why it isn't instead:

$SE = \sqrt{\frac{p(1- p)}{n}}$

To me it seems that both can make sense in some way. In the first case, we are constructing a null hypothesis of 900 / 1000, which we would expect to get if we sampled 1000 people. This is most consistent with the idea of "Given the null is true, what is the probability you will observe a deviation as large or larger than the sample mean?". But the downside is that if I now claim that 99999/100000 dentists recommend the tooth paste, my SE is essentially 0 since plugging in 0.99999 reduces the first equation, so rejecting the null becomes difficult. In other words, I can arbitrarily reduce the SE, and what's more, as I make the claim more extreme and more precise, it becomes harder to reject. This seems counter-intuitive.

In the second case we are saying that the error is in our sample mean, and treating the null as a point estimate. In this case, the only way to reduce the SE is to increase my sample, which makes intuitive sense - as my observed estimate of the sample mean becomes more precise, I can more easily reject a given claim. But the confusing thing here is that since SE depends on p, then computing the likelihood of seeing p "If the null hypothesis, p_o, is true" doesn't work, since the SE will be different at p_o.

I guess the bottom line is that I'm finding binomial stats a bit unintuitive. Since these methods give different SE and have these interesting properties, I feel like I'm missing something on why the first way is how I see people compute SE in the case of a z-test.

$\endgroup$
5
  • $\begingroup$ The null hypothesis provides a particular distribution to be used in the test. Under $H_0$ we have $X \sim BINOM(1000, 0.9),$ so $E(\hat p) = E(X/1000) = p_0 = 0.9.$ Etc. $\endgroup$
    – BruceET
    Apr 11, 2019 at 0:48
  • $\begingroup$ Right, as I said it makes sense from the Hypothesis Testing perspective. Despite that it is the correct way to compute things, it still continues to be unintuitive to me that it has the properties it does, vs the properties of the second method which seem more like what you'd want. I'm not saying the second method is correct, just that it seems to have better properties, and so I'm wondering about that. $\endgroup$
    – CHP
    Apr 11, 2019 at 1:14
  • $\begingroup$ See my Answer for comparison with the exact binomial test. Maybe what's mixing you up is that the usual 95% CI uses $\hat p$ to estimate $p,$ because there is no value of $p$ provided by a null hypothesis. $\endgroup$
    – BruceET
    Apr 11, 2019 at 1:33
  • 1
    $\begingroup$ Where you have p(p-1) it should be p(1-p). $\endgroup$
    – Glen_b
    Apr 11, 2019 at 14:44
  • $\begingroup$ Right, fixed! I was computing it correctly in practice so that wasn't the confusion (see comment in the accepted answer), but don't want to spread more confusion with bad equations :) $\endgroup$
    – CHP
    Apr 11, 2019 at 16:07

1 Answer 1

1
$\begingroup$

Exact binomial test. If you are testing $H_0: p = .9$ vs. $H_a: p < .9$ (one-sided test) then an exact test uses critical value $c = 883$ at level $\alpha \le .05.$

Under $H_0,$ we have $X \sim \mathsf{Binom}(n = 1000, p_0=.9),$ with $P(X \le 883) = 0.0433 < 0.05,$ but $P(X \le 884) = 0.0535 > 0.05.$ We reject $H_0$ based on your survey, which gave $X = 700 < c = 883.$ The P-value is $< 0.0001.$

Computations using R:

qbinom(.05, 1000, .9)
[1] 884
pbinom(884, 1000, .9)
[1] 0.05345008
pbinom(883, 1000, .9)
[1] 0.04334389
pbinom(700, 1000, .9)
[1] 6.823349e-69

The figure below shows the PDF of $\mathsf{Binom}(1000, 0.9);$ the vertical red line is at $c;$ the observed value $X = 700$ is off the graph to the left.

enter image description here

Normal test of binomial proportion. If you want to use the normal approximation, then the we have test statistic

$$Z = \frac{\hat p - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} = -21.08,$$ and we reject $H_0$ at the 5% level because $Z < - 1.645.$ Because $Z$ is approximately standard normal the P-value is smaller than 0.0001.$

p.0 = .9; p.hat = .7; n = 1000
se = sqrt(p.0*(1-p.0)/n); se
[1] 0.009486833
z = (p.hat - p.0)/se;  z
[1] -21.08185
pnorm(z)
[1] 5.836396e-99

The figure shows the standard normal density curve along with the critical value for a left-sided test at level 5%.

enter image description here

$\endgroup$
1
  • $\begingroup$ Argh I think it's just me being confused by the setup of the problem itself, and the fact that the SE is a function of the effect. Of course as the SE gets smaller it's easier, not harder to reject the null. In the standard normal distribution case, the SE doens't change whether it's at the location of the null or the location of the observed data - it's just a function of N. So this property of changing SE based on p was throwing me. $\endgroup$
    – CHP
    Apr 11, 2019 at 5:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.