# Using same sample in two successive hypothesis tests

Consider, for example, the problem of testing the significance of a difference between two percentages or proportions; a sample of $$n_1$$ contains $$t_1$$ individuals with a given character and an independent sample of $$n_2$$ contains $$t_2$$. Following the rule (1), the standard error of the difference $$d = t_1/n_1 - t_2/n_2$$ is often given as $$\sigma_d = \sqrt{\frac{t_1}{n_1^2}\left(1-\frac{t_1}{n_1}\right) - \frac{t_2}{n_2^2}\left(1-\frac{t_2}{n_2}\right)}$$

But in using $$\frac{t_1}{n_1^2}\left(1-\frac{t_1}{n_1}\right)$$ and $$\frac{t_2}{n_2^2}\left(1-\frac{t_2}{n_2}\right)$$ as the squares of the estimates of the two standard errors, we are proceeding on the supposition that sample estimates must be made of two different population proportions $$p_1$$ and $$p_2$$. Actually, it is desired to test the hypothesis that $$p_1 = p_2 = p$$, and it follows that the best estimate of $$p$$ is obtained by combing together the two samples, whence we obtain

Estimate of $$\sigma_d = \sqrt{\frac{t_1+t_2}{n_1+n_2}\left(1 - \frac{t_1+t_2}{n_1+n_2}\right)\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}$$

This doesn't seem to be right, as the same data ($$t_1, n_1, t_2, n_2$$) is used in both

1. the hypothesis that $$p_1 = p_2 = p$$, and
2. some unnamed hypothesis about the value of $$p$$. I presume there is such a test, since an estimation of $$p$$ is implied ($$\frac{t_1 + t_2}{n_1 + n_2}$$), the standard error of which is also estimated.

Why is it OK to re-use the same data in two successive hypothesis tests? The feels like $$p$$-hacking, though I cannot prove it.

Here is a clarification about the second hypothesis. While the paper states that it considers "the problem of testing the significance of a difference between two percentages or proportions", I think it more or less also considers the confidential interval of $$p$$, since both the value and standard error of $$p$$ are given. As far as I know, constructing the confidential interval of $$p$$ is equivalent to testing a hypothesis about the value of $$p$$.

You may disagree with me on this point, and honestly, I also find it kind of weak. However, this doesn't change the nature of this question. Feel free to consider Neyman and Pearson's inference to be a hypothesis test followed by an estimation of $$p$$ using the same data. My point is, the same data is used in twice in successive statistical procedures, and the latter explicitly depends on the former.

• Could you give more details about unnamed hypothesis? I did not figure out what it is. – user158565 Jul 15 at 6:30
• @user158565 Sorry for the confusion. I have edited the question to provide more explanation. – nalzok Jul 15 at 7:33
• I still cannot construct the second null hypothesis. – user158565 Jul 15 at 21:30