Timeline for Can you multiply p-values if you perform the same test multiple times?
Current License: CC BY-SA 4.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 11, 2020 at 10:41 | history | edited | Christian Hennig | CC BY-SA 4.0 |
added 2 characters in body
|
| Oct 11, 2020 at 1:28 | vote | accept | James Ronald | ||
| Oct 10, 2020 at 21:19 | comment | added | Christian Hennig | Note that if you run two tests and pick the larger $T$, this is different from just running one test, and you need to correct for multiple testing. You can't just try to find significance until you have a large enough $T$. This is where Bonferroni comes in. So running a second test can only make your result "more significant" if the second p-value is less than half of the first one. The proper way to use "more independent observations" than used for $T_1$ would be to define a single aggregated test, rather than combining the two separate p-values; Fisher's method is one way of doing that. | |
| Oct 10, 2020 at 21:13 | comment | added | Christian Hennig | First question: Not quite. My point is that if you have two tests, there is no trivial definition of what exactly the outcomes are that are "less in line with the $H_0$ than what was observed". One can argue that $\{T_1 \ge t_1\}\cap \{T_2\ge t_2\}$ will not cover them all, whereas the union set covers too much. There is no such thing as a single well defined "correct" p-value in this case. Obviously $P\{T_1\ge t_1\}$ is still the p-value of the first test, and it's the same as running the second test and ignoring it, but why would you want to do that? | |
| Oct 10, 2020 at 21:06 | comment | added | Christian Hennig | Re Bonferroni: 2min means twice the minimum, correct. Wikipedia writes about Bonferroni that $p_i\le \alpha/m$ rejects the null hypothesis. $m$ is 2 here, and if all $p_i$ test the same null hypothesis, you can just consider the minimum, so comparing the minimum $p_i$ to $\alpha/2$ is just the same as having a p-value $2\min p$ (that for testing at $\alpha$-level has to be compared with $\alpha$). | |
| Oct 10, 2020 at 20:59 | comment | added | James Ronald | Also, in the Bonferroni corrected p-value equation you gave, what does the $2min$ notation mean? Or does that just mean multiplying the minimum of those two probabilities by by $2$? The equation seems to be different on Wikipedia | |
| Oct 10, 2020 at 20:53 | comment | added | James Ronald | Thank you for the detailed answer, I think I understand. Just to make sure I understand: so problem with getting the probability of $ \{ { T1\geq t1} \cap {T2 \geq t2 } \} $ is that it will not include all of the observations that are less than $T1$ but greater than $T2$ (assuming $T1 \geq T2$), even though these should be included? If so, then is the correct p-value always just equal to $ P( \{ T1 \geq t1 \} )$ (again assuming $T1 \geq T2$)? | |
| Oct 10, 2020 at 15:08 | history | edited | Christian Hennig | CC BY-SA 4.0 |
added 47 characters in body
|
| Oct 10, 2020 at 10:37 | history | edited | Christian Hennig | CC BY-SA 4.0 |
added 92 characters in body
|
| Oct 10, 2020 at 10:26 | history | edited | Christian Hennig | CC BY-SA 4.0 |
added 9 characters in body
|
| Oct 10, 2020 at 10:06 | history | edited | Christian Hennig | CC BY-SA 4.0 |
added 791 characters in body
|
| Oct 10, 2020 at 9:44 | history | answered | Christian Hennig | CC BY-SA 4.0 |