1
$\begingroup$

Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $S_1$ from population $P$, to test a hypothesis $\mathcal{H}_0$; I obtain a p-value $p_1 = 0.075$. Since I really want to publish a paper, I decide to run another trial on a new i.i.d. sample $S_2$ drawn from the same population, and obtain the second p-value $p_2$. At this point I have to stop, because there is no time to run further trials.

If the number of trials (two) was pre-determined, I would either calcualate Fisher's combined p-value or simply use the Bonferroni correction. But in my case it is random and data-dependent. Is there any statistically sound method I can combine $p_1$ and $p_2$ into a single p-value to test $\mathcal{H}_0$? Or is it simply p-hacking?

$\endgroup$
  • 2
    $\begingroup$ You might want to look into sequential analysis $\endgroup$ – Glen_b Jul 29 '19 at 22:56
1
$\begingroup$

The concept of $\alpha$-value is that you have a certain amount of probability mass that you can "use up". For instance, suppose you're trying to test whether a coin is fair, and you flip it 10 times. You might say "Well, the probability of it coming up heads 10 times, given that it's fair, is 1/1000. So if I reject the null in that case, I've used up 0.001 of $\alpha$." And if you don't have any reason to suspect that it's weighted towards heads rather than tails, then you should be doing a two-tailed test, so now you've "used up" $0.002$ of $\alpha$. You keep on going choosing rejection cases until you've used up all of $\alpha$. Since you don't want to be accused of cherry-picking the cases, you should probably take them from "most extreme" to "least extreme": in this case, that means first the case of zero tails/zero heads, then one tails/one heads, etc. If your $\alpha$-value is $0.05$, and you claim that your rejection region consists only of exactly two tails, then technically this is a valid test, but if it comes up two tails, it's going to look incredibly suspicious that your rejection region consists only of the results you happened to get.

If your $\alpha$ is $0.05$, and you're willing to stop after one sample if the $p$-value is less than $0.05$, then you've already used all of your $\alpha$. You have none left for your second sample. There's simply no legitimate way of combining $p$-values to get less than $0.05$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Got it. Thanks. What if my p-values are conservative? Does it give me more leeway? $\endgroup$ – quant_dev Jul 30 '19 at 10:29
  • $\begingroup$ @quant_dev If your $\alpha$ is $0.05$, and you reject the null only if the $p$-value for the first sample is less than $0.02$, then you've used up only $0.02$ of your $\alpha$. So you can have a policy of rejected the null based on the second sample, as long as the probability of your policy rejecting the null is less than $0.03$. $\endgroup$ – Acccumulation Jul 30 '19 at 14:57
  • $\begingroup$ I wish I could accept two answers to my question :) $\endgroup$ – quant_dev Jul 30 '19 at 16:30
2
$\begingroup$

Sorry I was not aware that you have the early stopping rule. As @Accumulation answered, your rule already use all $\alpha$ budget. If you decide whether you reject the null or not based on the second p-value only, your rejection rule is to reject the null if $(p_1, p_2) \in [0, \alpha_1] \times [0,1] \cup [0,1] \times [0, \alpha_2]$ where $\alpha_1$ is the first threshold you used to decide stop or continue and $\alpha_2$ is the threshold for the second p-value. In this case, it indues $\alpha_1 + \alpha_2 - \alpha_1 \alpha_2 = \alpha_1 + \alpha_2 (1- \alpha_1) \leq \alpha_1 + \alpha_2$ size test. This is the reson why I said all your $\alpha$ budget was used.

In this case, your combined p-value is given as $$ p = p_1 I(p_1 \leq \alpha_1) + \left\{\alpha_1 + p_2(1-\alpha_1)\right\}I(p_1 > \alpha_1). $$

Therefore, once $p_1 > \alpha_1$, your p-value is greater than or equal to $\alpha_1$ regardless of $p_2$ value.

==========================================================

[Previous answer - I was not aware the stopping rule]

I assume you are working with a non-conservative null, that is, under the null, the p-value has the uniform distribution on $[0,1]$. Since your p-values are two independent samples from $U[0,1]$ under the null, you can use any "data-independent" rejection region on $[0,1]^2$ whose area is less than your test level $\alpha$. (Most natural one would be $[0, \sqrt{\alpha}]^2$ and, in this case, p-value is $(\max\{p_1, p_2\})^2$)

However, if you can calculate a new p-value based on $S_1 \cup S_2$, you can also use this p-value as a merged p-value since you stopped trials in a data-independent manner (out-of-time).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Re 1st paragraph: does it matter that the 2nd p-value was calculated conditioned on the 1st being below a certain threshold? (I.e. if $p_1 = 0.0001$, I will stop after the 1st trial). Re 2nd paragraph: what if I didn't record the data? (E.g. the dataset is too big). $\endgroup$ – quant_dev Jul 30 '19 at 10:23
  • 1
    $\begingroup$ Sorry I was not aware that you have the early stopping rule. As @Accumulation answered, your rule already use all $\alpha$ budget. You can imagine that your rejection rule is to reject the null if $ (p_1, p_2) \in [0, \alpha_1] \times [0,1] \cup [0,1] \times [0, \alpha_2]$ where $\alpha_1$ is the first threshold you used to decide stop or continue and $\alpha_2$ is the threshold for the second p-value. In this case, it indues $\alpha_1 + \alpha_2 - \alpha_1 \alpha_2 = \alpha_1 + \alpha_2(1-\alpha_2)$ size test. $\endgroup$ – JaeHyeok Shin Jul 30 '19 at 17:55
  • 1
    $\begingroup$ I edited my answer due to the space limit in the comment. $\endgroup$ – JaeHyeok Shin Jul 30 '19 at 18:13
1
$\begingroup$

One method you can use, which would lose power quickly, is to generalize Bonferroni correction. That is, decide on correction / penalty $m_n$ to apply to each test, such that $\sum_{n=1}^{\infty} m_n \le 1$. One possible choice is $m_n = 2^{-n}$.

Assume you perform unlimited data adaptive tests $T_1, T_2, \ldots$, where each test depends on the tests before it. For each test statistics $X_n(T_1,\ldots,T_n)$ consider a threshold $\beta_{n}$ s.t. $$ P_{T_1,\ldots,T_n}(X_n > \beta_{n}) < \frac{\alpha}{m_{n}} $$ for a desired $\alpha$ error rate, and the probability is over the data of all the tests before $T_i$.

Then, we have

$$ P_{T_1,T_2, \ldots}\left(\exists n ,X_n \ge \beta_{n}\right) \le \sum_{n=1}^{\infty} P_{T_1, T_2, \ldots}\left(X_n \ge \beta_{n} \right) = \sum_{n=1}^{\infty} P_{T_1,\ldots,T_n}\left(X_n \ge \beta_{n} \right) \le \sum_{n=1}^\infty \frac{\alpha}{m_{n}} \le \alpha $$

Note that it event doesn't matter when you stop, as long as you chose $m_{n}$ ahead of time.

Note that there might be a better method to exploit the structure of the problem, and gain more liberal thresholds.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. Is this related to the Holm-Bonferroni method? $\endgroup$ – quant_dev Jul 30 '19 at 10:32
  • 1
    $\begingroup$ It is related to plain Bonfferoni coresection. Holm-Bonferonni is a bit different, and I dont see how it could be adapted to potentially infinite tests. $\endgroup$ – tmrlvi Jul 30 '19 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.