Combining p-values when the trials are independent but their number is data-dependent and random

Question

Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $S_1$ from population $P$, to test a hypothesis $\mathcal{H}_0$; I obtain a p-value $p_1 = 0.075$. Since I really want to publish a paper, I decide to run another trial on a new i.i.d. sample $S_2$ drawn from the same population, and obtain the second p-value $p_2$. At this point I have to stop, because there is no time to run further trials.

If the number of trials (two) was pre-determined, I would either calcualate Fisher's combined p-value or simply use the Bonferroni correction. But in my case it is random and data-dependent. Is there any statistically sound method I can combine $p_1$ and $p_2$ into a single p-value to test $\mathcal{H}_0$? Or is it simply p-hacking?

Answer 1

The concept of $\alpha$-value is that you have a certain amount of probability mass that you can "use up". For instance, suppose you're trying to test whether a coin is fair, and you flip it 10 times. You might say "Well, the probability of it coming up heads 10 times, given that it's fair, is 1/1000. So if I reject the null in that case, I've used up 0.001 of $\alpha$." And if you don't have any reason to suspect that it's weighted towards heads rather than tails, then you should be doing a two-tailed test, so now you've "used up" $0.002$ of $\alpha$. You keep on going choosing rejection cases until you've used up all of $\alpha$. Since you don't want to be accused of cherry-picking the cases, you should probably take them from "most extreme" to "least extreme": in this case, that means first the case of zero tails/zero heads, then one tails/one heads, etc. If your $\alpha$-value is $0.05$, and you claim that your rejection region consists only of exactly two tails, then technically this is a valid test, but if it comes up two tails, it's going to look incredibly suspicious that your rejection region consists only of the results you happened to get.

If your $\alpha$ is $0.05$, and you're willing to stop after one sample if the $p$-value is less than $0.05$, then you've already used all of your $\alpha$. You have none left for your second sample. There's simply no legitimate way of combining $p$-values to get less than $0.05$.

Answer 2

Sorry I was not aware that you have the early stopping rule. As @Accumulation answered, your rule already use all $\alpha$ budget. If you decide whether you reject the null or not based on the second p-value only, your rejection rule is to reject the null if $(p_1, p_2) \in [0, \alpha_1] \times [0,1] \cup [0,1] \times [0, \alpha_2]$ where $\alpha_1$ is the first threshold you used to decide stop or continue and $\alpha_2$ is the threshold for the second p-value. In this case, it indues $\alpha_1 + \alpha_2 - \alpha_1 \alpha_2 = \alpha_1 + \alpha_2 (1- \alpha_1) \leq \alpha_1 + \alpha_2$ size test. This is the reson why I said all your $\alpha$ budget was used.

In this case, your combined p-value is given as $$ p = p_1 I(p_1 \leq \alpha_1) + \left\{\alpha_1 + p_2(1-\alpha_1)\right\}I(p_1 > \alpha_1). $$

Therefore, once $p_1 > \alpha_1$, your p-value is greater than or equal to $\alpha_1$ regardless of $p_2$ value.

==========================================================

[Previous answer - I was not aware the stopping rule]

I assume you are working with a non-conservative null, that is, under the null, the p-value has the uniform distribution on $[0,1]$. Since your p-values are two independent samples from $U[0,1]$ under the null, you can use any "data-independent" rejection region on $[0,1]^2$ whose area is less than your test level $\alpha$. (Most natural one would be $[0, \sqrt{\alpha}]^2$ and, in this case, p-value is $(\max\{p_1, p_2\})^2$)

However, if you can calculate a new p-value based on $S_1 \cup S_2$, you can also use this p-value as a merged p-value since you stopped trials in a data-independent manner (out-of-time).

Answer 3

One method you can use, which would lose power quickly, is to generalize Bonferroni correction. That is, decide on correction / penalty $m_n$ to apply to each test, such that $\sum_{n=1}^{\infty} m_n \le 1$. One possible choice is $m_n = 2^{-n}$.

Assume you perform unlimited data adaptive tests $T_1, T_2, \ldots$, where each test depends on the tests before it. For each test statistics $X_n(T_1,\ldots,T_n)$ consider a threshold $\beta_{n}$ s.t. $$ P_{T_1,\ldots,T_n}(X_n > \beta_{n}) < \frac{\alpha}{m_{n}} $$ for a desired $\alpha$ error rate, and the probability is over the data of all the tests before $T_i$.

Then, we have

$$ P_{T_1,T_2, \ldots}\left(\exists n ,X_n \ge \beta_{n}\right) \le \sum_{n=1}^{\infty} P_{T_1, T_2, \ldots}\left(X_n \ge \beta_{n} \right) = \sum_{n=1}^{\infty} P_{T_1,\ldots,T_n}\left(X_n \ge \beta_{n} \right) \le \sum_{n=1}^\infty \frac{\alpha}{m_{n}} \le \alpha $$

Note that it event doesn't matter when you stop, as long as you chose $m_{n}$ ahead of time.

Note that there might be a better method to exploit the structure of the problem, and gain more liberal thresholds.