Can you multiply p-values if you perform the same test multiple times?

Question

I believe the interpretation of the p-value is that it is the probability of seeing your sample's test statistic under the null hypothesis.

But what happens if you perform the same exact test multiple times and get multiple p-values? Could you use the multiplication rule from probability to multiply the first p-value by the second, thus getting a new overall probability of seeing your test statistic?

e.g. you do a t-test and get a p-value of 0.05, and then you perform the same test with a completely different sample and get a p-value of 0.10. In this case, the probability of seeing those two test statistics under the null hypothesis would be $0.05 \times 0.1 = 0.005$, which is a less likely and thus more significant value?

Answer 1

"I believe the interpretation of the p-value is that it is the probability of seeing your sample's test statistic under the null hypothesis."

No. It is the probability to see your sample's test statistic or something that is even less in line with the null hypothesis ($H_0$) under the $H_0$, which I write as $P_0\{T\ge t\}$, where $T$ is the test statistic and $t$ is its observed value, assuming here that a large value of $T$ provides evidence against $H_0$ (the argument can as well be made for the $\{T\le t\}$ or the two-sided case).

If you have, say, $p=0.06$ in one test $T_1$ with result $t_1$ and $p=0.6$ in the next ($T_2, t_2$; let's assume they were done on independent observations), if you multiply these two, what you get is the probability of $\{T_1\ge t_1\} \cap \{T_2\ge t_2\}$, i.e., the probability that $T_1$ and $T_2$ are large under the $H_0$. This is of course less likely than having at least one of them large. But there are cases with at least one of them large that count at least as strongly against the $H_0$, such as having $T_1$ extremely large even if $T_2$ doesn't indicate problems with the $H_0$, so the event $\{T_1\ge t_1\} \cap \{T_2\ge t_2\}$, of which you get the probability by multiplying the p-values, does not cover all possibilities to observe something that is even less in line with the $H_0$ than what you observed, and is therefore smaller than a valid "combined" p-value would need to be.

In my example above, surely after having observed $t_1$ with $P_0\{T_1\ge t_1\}=0.06$, observing $t_2$ with $P_0\{T_2\ge t_2\}=0.6$ doesn't make the overall result indicate any stronger against the $H_0$ (as multiplying the p-values would suggest), because observing something with $P_0\{T_2\ge t_2\}=0.6$ is perfectly reasonable under $H_0$; however observing $T_1$ even larger than $t_1$ would arguably count stronger against $H_0$ even with observing a smaller $T_2$.

The problem with combining p-values from more than one test is that if you only have a one-dimensional test statistic, as long as this statistic is suitably defined, it is clear how you can find all possible outcomes that are less in line with $H_0$ than your observation (depending on the test statistic either by looking at all larger, or all smaller values, or combining the two sides). However, with two or more values of the test statistic, in the higher dimensional space of possible outcomes it is much more difficult to define what "less in line with $H_0$" actually means. One possibility to play it safe is to look at $P_0(\{T_1\ge t_1\}\cup\{T_2\ge t_2\})$, the probability that at least one of $T_1$ and $T_2$ is too large. This for sure covers all possibilities that the pair $(T_1,T_2)$ is less in line with $H_0$ than the observations $(t_1,t_2)$. It actually covers far too much and is therefore very conservative. It may in fact be seen as useless, because its probability will always be bigger than $P_0\{T_1\ge t_1\}$, so this won't allow you to find a significance based on $(T_1,T_2)$ if you don't find one based on $T_1$ alone. If the two tests are independent, as apparently assumed here, $P_0(\{T_1\ge t_1\}\cup\{T_2\ge t_2\})=1-(1-P_0\{T_1\ge t_1\})(1-P_0\{T_2\ge t_1\})=0.624$ in the example, so there you have your multiplication.

Note that $2\min(P_0\{T_1\ge t_1\},P_0\{T_2\ge t_2\})=0.12$ in the example is the so-called Bonferroni-corrected p-value, which gives an upper bound on the probability that any of the two indicates at least as much against $H_0$ than the one that has the stronger indication, which is somewhat better than $P_0(\{T_1\ge t_1\}\cup\{T_2\ge t_2\})$, but still will not allow you to have an overall combined p-value that is smaller than all those you observe for the isolated tests. Under independence this can be improved to $1-(1-\min (P_0\{T_1\ge t_1\},P_0\{T_2\ge t_1\}))^2=0.116$, not much change here. (Edit: Fisher's method as linked in the answer of gunes will normally be better than this in the independence case.)

Answer 2

No, because then you could repeat any insignificant p-value and get a significant result, e.g. $0.9^{100}\approx 0.0000027$. Fisher's method is one way to combine multiple p-values.

Answer 3

Let $X \sim N(\mu,\sigma^2)$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Now,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

Clearly, the two expressions are not same.

Now if I understand your question, you want to know what if we reject null based on $p_xp_y$ as compared to when we reject based on $p_{xy}$.

Under a true null hypothesis, p-value is uniformly distributed on $[0,1]$. So,

$$Pr(p_{xy} \leq 0.05) = 0.05$$

However, same is not true for $p_xp_y$ as it is not uniformly distributed. It's cdf is $z-z\ln{z}$. See this for derivation.

$$Pr(p_xp_y \leq 0.05) \approx 0.2$$

So you are clearly rejecting the null hypothesis much more often leading to higher type I error.

As such, I didn't need to assume anything about the distribution or test statistic. That was done just to illustrate that the two expressions are not the same.

The second part of the answer holds always because:

$$z-z\ln{z} > z \ \ \ \ \forall z \in [0,1)$$

Answer 4

There are many ways to combine $p$-values, in fact we even have a tag for them combining-p-values. One method which is often used is Fisher's method which does, in effect, multiply the $p$-values. In fact logs are taken and summed which comes to the same thing. The crucial difference though is that this does not yield the new $p$-value directly but rather it yields a $\chi^2$ value from which the new $p$-value can be obtained.

Answer 5

You can, I suppose, use a product of $p$-values $P \equiv \prod p_i$ as a measure of evidence if you like.

But it isn't itself a $p$-value, as for one thing it isn't uniformly distributed between 0 and 1 under the null, i.e., we don't have $P \sim U(0, 1)$ under the null. So it cannot be used to control type-1 errors in the usual manner. If we reject when $P \le \alpha$, we won't get a type-1 error rate of $\alpha$.

Answer 6

Could you use the multiplication rule from probability to multiply the first p-value by the second, thus getting a new overall probability of seeing your test statistic?

e.g. you do a t-test and get a p-value of 0.05, and then you perform the same test with a completely different sample and get a p-value of 0.10. ...
[p-value= $ 0,05 * 0,1 = 0,005$?]

No, it is wrong. Even if the two sample are disjoint, inference is not so simple. The answer of gunes give us an interesting intuition. Another way for convincing you is that you can join the two sample and perform the same test. If the rule that you assume was correct, the result should be the same. If the $H_0$ is true, the p-value should increase with the number of data, otherwise it should decrease with it. Following your rule p-value decrease even if $H_0$ is true.