Quoting On the Problem of the Most Efficient Tests of Statistical Hypotheses (J. Neyman; E. S. Pearson, 1933)

Consider, for example, the problem of testing the significance of a difference between two percentages or proportions; a sample of $n_1$ contains $t_1$ individuals with a given character and an independent sample of $n_2$ contains $t_2$. Following the rule (1), the standard error of the difference $d = t_1/n_1 - t_2/n_2$ is often given as $$\sigma_d = \sqrt{\frac{t_1}{n_1^2}\left(1-\frac{t_1}{n_1}\right) - \frac{t_2}{n_2^2}\left(1-\frac{t_2}{n_2}\right)} $$

But in using $\frac{t_1}{n_1^2}\left(1-\frac{t_1}{n_1}\right)$ and $\frac{t_2}{n_2^2}\left(1-\frac{t_2}{n_2}\right)$ as the squares of the estimates of the two standard errors, we are proceeding on the supposition that sample estimates must be made of two different population proportions $p_1$ and $p_2$. Actually, it is desired to test the hypothesis that $p_1 = p_2 = p$, and it follows that the best estimate of $p$ is obtained by combing together the two samples, whence we obtain

Estimate of $$\sigma_d = \sqrt{\frac{t_1+t_2}{n_1+n_2}\left(1 - \frac{t_1+t_2}{n_1+n_2}\right)\left(\frac{1}{n_1} + \frac{1}{n_2}\right)} $$

This doesn't seem to be right, as the same data ($t_1, n_1, t_2, n_2$) is used in both

  1. the hypothesis that $p_1 = p_2 = p$, and
  2. some unnamed hypothesis about the value of $p$. I presume there is such a test, since an estimation of $p$ is implied ($\frac{t_1 + t_2}{n_1 + n_2}$), the standard error of which is also estimated.

Why is it OK to re-use the same data in two successive hypothesis tests? The feels like $p$-hacking, though I cannot prove it.

Here is a clarification about the second hypothesis. While the paper states that it considers "the problem of testing the significance of a difference between two percentages or proportions", I think it more or less also considers the confidential interval of $p$, since both the value and standard error of $p$ are given. As far as I know, constructing the confidential interval of $p$ is equivalent to testing a hypothesis about the value of $p$.

You may disagree with me on this point, and honestly, I also find it kind of weak. However, this doesn't change the nature of this question. Feel free to consider Neyman and Pearson's inference to be a hypothesis test followed by an estimation of $p$ using the same data. My point is, the same data is used in twice in successive statistical procedures, and the latter explicitly depends on the former.

  • $\begingroup$ Could you give more details about unnamed hypothesis? I did not figure out what it is. $\endgroup$ – user158565 Jul 15 at 6:30
  • $\begingroup$ @user158565 Sorry for the confusion. I have edited the question to provide more explanation. $\endgroup$ – nalzok Jul 15 at 7:33
  • $\begingroup$ I still cannot construct the second null hypothesis. $\endgroup$ – user158565 Jul 15 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.