# Can you multiply p-values if you perform the same test multiple times?

I believe the interpretation of the p-value is that it is the probability of seeing your sample's test statistic under the null hypothesis.

But what happens if you perform the same exact test multiple times and get multiple p-values? Could you use the multiplication rule from probability to multiply the first p-value by the second, thus getting a new overall probability of seeing your test statistic?

e.g. you do a t-test and get a p-value of 0.05, and then you perform the same test with a completely different sample and get a p-value of 0.10. In this case, the probability of seeing those two test statistics under the null hypothesis would be $$0.05 \times 0.1 = 0.005$$, which is a less likely and thus more significant value?

• This isn't a valid p-value because it doesn't correspond to a test statistic that orders & partitions the sample space. Shade in some different events of which you're calculating the probability on a plot of the sample space, & you'll see the nesting requirement for rejection regions is violated. – Scortchi - Reinstate Monica Oct 10 '20 at 9:06

"I believe the interpretation of the p-value is that it is the probability of seeing your sample's test statistic under the null hypothesis."

No. It is the probability to see your sample's test statistic or something that is even less in line with the null hypothesis ($$H_0$$) under the $$H_0$$, which I write as $$P_0\{T\ge t\}$$, where $$T$$ is the test statistic and $$t$$ is its observed value, assuming here that a large value of $$T$$ provides evidence against $$H_0$$ (the argument can as well be made for the $$\{T\le t\}$$ or the two-sided case).

If you have, say, $$p=0.06$$ in one test $$T_1$$ with result $$t_1$$ and $$p=0.6$$ in the next ($$T_2, t_2$$; let's assume they were done on independent observations), if you multiply these two, what you get is the probability of $$\{T_1\ge t_1\} \cap \{T_2\ge t_2\}$$, i.e., the probability that $$T_1$$ and $$T_2$$ are large under the $$H_0$$. This is of course less likely than having at least one of them large. But there are cases with at least one of them large that count at least as strongly against the $$H_0$$, such as having $$T_1$$ extremely large even if $$T_2$$ doesn't indicate problems with the $$H_0$$, so the event $$\{T_1\ge t_1\} \cap \{T_2\ge t_2\}$$, of which you get the probability by multiplying the p-values, does not cover all possibilities to observe something that is even less in line with the $$H_0$$ than what you observed, and is therefore smaller than a valid "combined" p-value would need to be.

In my example above, surely after having observed $$t_1$$ with $$P_0\{T_1\ge t_1\}=0.06$$, observing $$t_2$$ with $$P_0\{T_2\ge t_2\}=0.6$$ doesn't make the overall result indicate any stronger against the $$H_0$$ (as multiplying the p-values would suggest), because observing something with $$P_0\{T_2\ge t_2\}=0.6$$ is perfectly reasonable under $$H_0$$; however observing $$T_1$$ even larger than $$t_1$$ would arguably count stronger against $$H_0$$ even with observing a smaller $$T_2$$.

The problem with combining p-values from more than one test is that if you only have a one-dimensional test statistic, as long as this statistic is suitably defined, it is clear how you can find all possible outcomes that are less in line with $$H_0$$ than your observation (depending on the test statistic either by looking at all larger, or all smaller values, or combining the two sides). However, with two or more values of the test statistic, in the higher dimensional space of possible outcomes it is much more difficult to define what "less in line with $$H_0$$" actually means. One possibility to play it safe is to look at $$P_0(\{T_1\ge t_1\}\cup\{T_2\ge t_2\})$$, the probability that at least one of $$T_1$$ and $$T_2$$ is too large. This for sure covers all possibilities that the pair $$(T_1,T_2)$$ is less in line with $$H_0$$ than the observations $$(t_1,t_2)$$. It actually covers far too much and is therefore very conservative. It may in fact be seen as useless, because its probability will always be bigger than $$P_0\{T_1\ge t_1\}$$, so this won't allow you to find a significance based on $$(T_1,T_2)$$ if you don't find one based on $$T_1$$ alone. If the two tests are independent, as apparently assumed here, $$P_0(\{T_1\ge t_1\}\cup\{T_2\ge t_2\})=1-(1-P_0\{T_1\ge t_1\})(1-P_0\{T_2\ge t_1\})=0.624$$ in the example, so there you have your multiplication.

Note that $$2\min(P_0\{T_1\ge t_1\},P_0\{T_2\ge t_2\})=0.12$$ in the example is the so-called Bonferroni-corrected p-value, which gives an upper bound on the probability that any of the two indicates at least as much against $$H_0$$ than the one that has the stronger indication, which is somewhat better than $$P_0(\{T_1\ge t_1\}\cup\{T_2\ge t_2\})$$, but still will not allow you to have an overall combined p-value that is smaller than all those you observe for the isolated tests. Under independence this can be improved to $$1-(1-\min (P_0\{T_1\ge t_1\},P_0\{T_2\ge t_1\}))^2=0.116$$, not much change here. (Edit: Fisher's method as linked in the answer of gunes will normally be better than this in the independence case.)

• Thank you for the detailed answer, I think I understand. Just to make sure I understand: so problem with getting the probability of $\{ { T1\geq t1} \cap {T2 \geq t2 } \}$ is that it will not include all of the observations that are less than $T1$ but greater than $T2$ (assuming $T1 \geq T2$), even though these should be included? If so, then is the correct p-value always just equal to $P( \{ T1 \geq t1 \} )$ (again assuming $T1 \geq T2$)? – James Ronald Oct 10 '20 at 20:53
• Also, in the Bonferroni corrected p-value equation you gave, what does the $2min$ notation mean? Or does that just mean multiplying the minimum of those two probabilities by by $2$? The equation seems to be different on Wikipedia – James Ronald Oct 10 '20 at 20:59
• Re Bonferroni: 2min means twice the minimum, correct. Wikipedia writes about Bonferroni that $p_i\le \alpha/m$ rejects the null hypothesis. $m$ is 2 here, and if all $p_i$ test the same null hypothesis, you can just consider the minimum, so comparing the minimum $p_i$ to $\alpha/2$ is just the same as having a p-value $2\min p$ (that for testing at $\alpha$-level has to be compared with $\alpha$). – Lewian Oct 10 '20 at 21:06
• First question: Not quite. My point is that if you have two tests, there is no trivial definition of what exactly the outcomes are that are "less in line with the $H_0$ than what was observed". One can argue that $\{T_1 \ge t_1\}\cap \{T_2\ge t_2\}$ will not cover them all, whereas the union set covers too much. There is no such thing as a single well defined "correct" p-value in this case. Obviously $P\{T_1\ge t_1\}$ is still the p-value of the first test, and it's the same as running the second test and ignoring it, but why would you want to do that? – Lewian Oct 10 '20 at 21:13
• Note that if you run two tests and pick the larger $T$, this is different from just running one test, and you need to correct for multiple testing. You can't just try to find significance until you have a large enough $T$. This is where Bonferroni comes in. So running a second test can only make your result "more significant" if the second p-value is less than half of the first one. The proper way to use "more independent observations" than used for $T_1$ would be to define a single aggregated test, rather than combining the two separate p-values; Fisher's method is one way of doing that. – Lewian Oct 10 '20 at 21:19

No, because then you could repeat any insignificant p-value and get a significant result, e.g. $$0.9^{100}\approx 0.0000027$$. Fisher's method is one way to combine multiple p-values.

• But why is that wrong? Why isn’t that equally strong evidence against the null hypothesis? I’m with you on your conclusion that we can’t do this, but I’m having more trouble than I’m proud to admit in articulating why. Maybe something to do with independence of the distributions of the test statistics? – Dave Oct 9 '20 at 23:46
• Because if we multiply, it'll be the probability of having a significant result (given $H_0$) $n$ times in row. But, we're interested in having a more reliable p-value by doing the test multiple times, i.e. a representative significance figure for an hypothetical experiment. – gunes Oct 9 '20 at 23:58
• To convince yourself that this answer is correct, you could try simulating 100 t-tests with a Normal sample with mean 0 (i.e. the null hypothesis is true): set.seed(102); r <- replicate(100,t.test(rnorm(100))\$p.value). Then look at the histogram hist(r) (which is approximately uniform between 0 and 1, as expected) and prod(r) which is tiny. – Ben Bolker Oct 10 '20 at 0:04
• Indeed, the concern with this is that you would seemingly be pulling statistical power out of thin air - it would allow you to split any sample you have into smaller and smaller samples that you perform tests on individually, the p-values of which you could multiply until you have a significant value. However, I would like to be able to justify why we cannot do this in terms of theory, as opposed to in terms of common sense. What is it about the logic presented in the post that does not hold? – James Ronald Oct 10 '20 at 0:14
• among other things, you're treating a p-value like a likelihood (i.e., probabilities of outcomes of multiple independent events should multiply), when it's not; it's a probability of the outcome being >= the observed statistic under the NH. – Ben Bolker Oct 10 '20 at 1:31

Let $$X \sim N(\mu,\sigma^2)$$ and $$\{x_i\}$$ and $$\{y_i\}$$ be two same size i.i.d samples from $$X$$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $$p_x$$ and $$p_y$$ be respective p-values for individual tests and $$p_{xy}$$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Now,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $$\Phi(.)$$ is the cdf for $$N(0,1)$$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

Clearly, the two expressions are not same.

Now if I understand your question, you want to know what if we reject null based on $$p_xp_y$$ as compared to when we reject based on $$p_{xy}$$.

Under a true null hypothesis, p-value is uniformly distributed on $$[0,1]$$. So,

$$Pr(p_{xy} \leq 0.05) = 0.05$$

However, same is not true for $$p_xp_y$$ as it is not uniformly distributed. It's cdf is $$z-z\ln{z}$$. See this for derivation.

$$Pr(p_xp_y \leq 0.05) \approx 0.2$$

So you are clearly rejecting the null hypothesis much more often leading to higher type I error.

As such, I didn't need to assume anything about the distribution or test statistic. That was done just to illustrate that the two expressions are not the same.

The second part of the answer holds always because:

$$z-z\ln{z} > z \ \ \ \ \forall z \in [0,1)$$

There are many ways to combine $$p$$-values, in fact we even have a tag for them . One method which is often used is Fisher's method which does, in effect, multiply the $$p$$-values. In fact logs are taken and summed which comes to the same thing. The crucial difference though is that this does not yield the new $$p$$-value directly but rather it yields a $$\chi^2$$ value from which the new $$p$$-value can be obtained.

You can, I suppose, use a product of $$p$$-values $$P \equiv \prod p_i$$ as a measure of evidence if you like.

But it isn't itself a $$p$$-value, as for one thing it isn't uniformly distributed between 0 and 1 under the null, i.e., we don't have $$P \sim U(0, 1)$$ under the null. So it cannot be used to control type-1 errors in the usual manner. If we reject when $$P \le \alpha$$, we won't get a type-1 error rate of $$\alpha$$.

Could you use the multiplication rule from probability to multiply the first p-value by the second, thus getting a new overall probability of seeing your test statistic?

e.g. you do a t-test and get a p-value of 0.05, and then you perform the same test with a completely different sample and get a p-value of 0.10. ...
[p-value= $$0,05 * 0,1 = 0,005$$?]

No, it is wrong. Even if the two sample are disjoint, inference is not so simple. The answer of gunes give us an interesting intuition. Another way for convincing you is that you can join the two sample and perform the same test. If the rule that you assume was correct, the result should be the same. If the $$H_0$$ is true, the p-value should increase with the number of data, otherwise it should decrease with it. Following your rule p-value decrease even if $$H_0$$ is true.