Combining p-values when the trials are independent but their number is data-dependent and random

Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $$S_1$$ from population $$P$$, to test a hypothesis $$\mathcal{H}_0$$; I obtain a p-value $$p_1 = 0.075$$. Since I really want to publish a paper, I decide to run another trial on a new i.i.d. sample $$S_2$$ drawn from the same population, and obtain the second p-value $$p_2$$. At this point I have to stop, because there is no time to run further trials.

If the number of trials (two) was pre-determined, I would either calcualate Fisher's combined p-value or simply use the Bonferroni correction. But in my case it is random and data-dependent. Is there any statistically sound method I can combine $$p_1$$ and $$p_2$$ into a single p-value to test $$\mathcal{H}_0$$? Or is it simply p-hacking?

• You might want to look into sequential analysis – Glen_b Jul 29 '19 at 22:56

The concept of $$\alpha$$-value is that you have a certain amount of probability mass that you can "use up". For instance, suppose you're trying to test whether a coin is fair, and you flip it 10 times. You might say "Well, the probability of it coming up heads 10 times, given that it's fair, is 1/1000. So if I reject the null in that case, I've used up 0.001 of $$\alpha$$." And if you don't have any reason to suspect that it's weighted towards heads rather than tails, then you should be doing a two-tailed test, so now you've "used up" $$0.002$$ of $$\alpha$$. You keep on going choosing rejection cases until you've used up all of $$\alpha$$. Since you don't want to be accused of cherry-picking the cases, you should probably take them from "most extreme" to "least extreme": in this case, that means first the case of zero tails/zero heads, then one tails/one heads, etc. If your $$\alpha$$-value is $$0.05$$, and you claim that your rejection region consists only of exactly two tails, then technically this is a valid test, but if it comes up two tails, it's going to look incredibly suspicious that your rejection region consists only of the results you happened to get.

If your $$\alpha$$ is $$0.05$$, and you're willing to stop after one sample if the $$p$$-value is less than $$0.05$$, then you've already used all of your $$\alpha$$. You have none left for your second sample. There's simply no legitimate way of combining $$p$$-values to get less than $$0.05$$.

• Got it. Thanks. What if my p-values are conservative? Does it give me more leeway? – quant_dev Jul 30 '19 at 10:29
• @quant_dev If your $\alpha$ is $0.05$, and you reject the null only if the $p$-value for the first sample is less than $0.02$, then you've used up only $0.02$ of your $\alpha$. So you can have a policy of rejected the null based on the second sample, as long as the probability of your policy rejecting the null is less than $0.03$. – Acccumulation Jul 30 '19 at 14:57
• I wish I could accept two answers to my question :) – quant_dev Jul 30 '19 at 16:30

Sorry I was not aware that you have the early stopping rule. As @Accumulation answered, your rule already use all $$\alpha$$ budget. If you decide whether you reject the null or not based on the second p-value only, your rejection rule is to reject the null if $$(p_1, p_2) \in [0, \alpha_1] \times [0,1] \cup [0,1] \times [0, \alpha_2]$$ where $$\alpha_1$$ is the first threshold you used to decide stop or continue and $$\alpha_2$$ is the threshold for the second p-value. In this case, it indues $$\alpha_1 + \alpha_2 - \alpha_1 \alpha_2 = \alpha_1 + \alpha_2 (1- \alpha_1) \leq \alpha_1 + \alpha_2$$ size test. This is the reson why I said all your $$\alpha$$ budget was used.

In this case, your combined p-value is given as $$p = p_1 I(p_1 \leq \alpha_1) + \left\{\alpha_1 + p_2(1-\alpha_1)\right\}I(p_1 > \alpha_1).$$

Therefore, once $$p_1 > \alpha_1$$, your p-value is greater than or equal to $$\alpha_1$$ regardless of $$p_2$$ value.

==========================================================

[Previous answer - I was not aware the stopping rule]

I assume you are working with a non-conservative null, that is, under the null, the p-value has the uniform distribution on $$[0,1]$$. Since your p-values are two independent samples from $$U[0,1]$$ under the null, you can use any "data-independent" rejection region on $$[0,1]^2$$ whose area is less than your test level $$\alpha$$. (Most natural one would be $$[0, \sqrt{\alpha}]^2$$ and, in this case, p-value is $$(\max\{p_1, p_2\})^2$$)

However, if you can calculate a new p-value based on $$S_1 \cup S_2$$, you can also use this p-value as a merged p-value since you stopped trials in a data-independent manner (out-of-time).

• Re 1st paragraph: does it matter that the 2nd p-value was calculated conditioned on the 1st being below a certain threshold? (I.e. if $p_1 = 0.0001$, I will stop after the 1st trial). Re 2nd paragraph: what if I didn't record the data? (E.g. the dataset is too big). – quant_dev Jul 30 '19 at 10:23
• Sorry I was not aware that you have the early stopping rule. As @Accumulation answered, your rule already use all $\alpha$ budget. You can imagine that your rejection rule is to reject the null if $(p_1, p_2) \in [0, \alpha_1] \times [0,1] \cup [0,1] \times [0, \alpha_2]$ where $\alpha_1$ is the first threshold you used to decide stop or continue and $\alpha_2$ is the threshold for the second p-value. In this case, it indues $\alpha_1 + \alpha_2 - \alpha_1 \alpha_2 = \alpha_1 + \alpha_2(1-\alpha_2)$ size test. – JaeHyeok Shin Jul 30 '19 at 17:55
• I edited my answer due to the space limit in the comment. – JaeHyeok Shin Jul 30 '19 at 18:13

One method you can use, which would lose power quickly, is to generalize Bonferroni correction. That is, decide on correction / penalty $$m_n$$ to apply to each test, such that $$\sum_{n=1}^{\infty} m_n \le 1$$. One possible choice is $$m_n = 2^{-n}$$.

Assume you perform unlimited data adaptive tests $$T_1, T_2, \ldots$$, where each test depends on the tests before it. For each test statistics $$X_n(T_1,\ldots,T_n)$$ consider a threshold $$\beta_{n}$$ s.t. $$P_{T_1,\ldots,T_n}(X_n > \beta_{n}) < \frac{\alpha}{m_{n}}$$ for a desired $$\alpha$$ error rate, and the probability is over the data of all the tests before $$T_i$$.

Then, we have

$$P_{T_1,T_2, \ldots}\left(\exists n ,X_n \ge \beta_{n}\right) \le \sum_{n=1}^{\infty} P_{T_1, T_2, \ldots}\left(X_n \ge \beta_{n} \right) = \sum_{n=1}^{\infty} P_{T_1,\ldots,T_n}\left(X_n \ge \beta_{n} \right) \le \sum_{n=1}^\infty \frac{\alpha}{m_{n}} \le \alpha$$

Note that it event doesn't matter when you stop, as long as you chose $$m_{n}$$ ahead of time.

Note that there might be a better method to exploit the structure of the problem, and gain more liberal thresholds.

• Thanks. Is this related to the Holm-Bonferroni method? – quant_dev Jul 30 '19 at 10:32
• It is related to plain Bonfferoni coresection. Holm-Bonferonni is a bit different, and I dont see how it could be adapted to potentially infinite tests. – tmrlvi Jul 30 '19 at 15:10