Skip to main content
Cross Validated

Return to Answer

removed misleading example
Source Link
Lasse Kliemann
Lasse Kliemann
  • 159
  • 3

Denote $p$ the $p$-value of your test (as a random variable) and fix some $\alpha$. Call a test result significant or positive when $p \leq \alpha$. We have $P(p \leq \alpha \,|\, H_0) \leq \alpha$. Moreover, let $\beta$ be such that $P(p > \alpha \,|\, H_1) \leq \beta$. Then $1-\beta$ is the power of the test.

Treating $H_0$ and $H_1$ as (complementary) events, Bayes' theorem gives: $$\frac{P(H_1 \, | \, p\leq\alpha)}{P(H_0 \,|\, p\leq\alpha)} = \frac{P(p\leq\alpha \,|\, H_1)}{P(p\leq\alpha \,|\, H_0)} \cdot \frac{P(H_1)}{P(H_0)} \geq \frac{1-\beta}{\alpha} \cdot \frac{P(H_1)}{P(H_0)}$$ This shows that the post odds for $H_1$ are a scaled version of the prior odds, with the strength of the scaling in favor for $H_1$ increasing with $1-\beta$. This means we learn more from a positive test when $1-\beta$ is large.

In particular, imagine a silly test with $1 - \beta = 0$. Then all significant results this test can produce are false-positives. So if you get a significant result from this silly test, it tells you nothing.

For further illustration, look at confidence intervals (CI). One may argue that larger sample size will make the CI more narrow and thus, if the test was significant for a smaller sample, it will also be significant for the larger sample. However, also the location of the CI can shift when we include more data in our sample, potentially making the result non-significant. It is also conceivable that the larger sample will have a much larger standard error and thus the CI will become wider in fact. One could say that a larger sample size gives the facts more opportunity to prove themselves.

There has been some interesting discussion lately about the interpretation of $p$-values, see, e.g.:

[1] Colquhoun, "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science, 2014

[2] Colquhoun, "The Reproducibility Of Research And The Misinterpretation Of P Values", 2017, http://www.biorxiv.org/content/early/2017/08/07/144337

[3] "What would Cohen say? A comment on $p < .005$", https://replicationindex.wordpress.com/2017/08/02/what-would-cohen-say-a-comment-on-p-005/

Concerning your particular result, I am not qualified to judge it. Using only your $p$-value and the classification from [2], it is between "weak evidence: worth another look" and "moderate evidence for a real effect".

Denote $p$ the $p$-value of your test (as a random variable) and fix some $\alpha$. Call a test result significant or positive when $p \leq \alpha$. We have $P(p \leq \alpha \,|\, H_0) \leq \alpha$. Moreover, let $\beta$ be such that $P(p > \alpha \,|\, H_1) \leq \beta$. Then $1-\beta$ is the power of the test.

Treating $H_0$ and $H_1$ as (complementary) events, Bayes' theorem gives: $$\frac{P(H_1 \, | \, p\leq\alpha)}{P(H_0 \,|\, p\leq\alpha)} = \frac{P(p\leq\alpha \,|\, H_1)}{P(p\leq\alpha \,|\, H_0)} \cdot \frac{P(H_1)}{P(H_0)} \geq \frac{1-\beta}{\alpha} \cdot \frac{P(H_1)}{P(H_0)}$$ This shows that the post odds for $H_1$ are a scaled version of the prior odds, with the strength of the scaling in favor for $H_1$ increasing with $1-\beta$. This means we learn more from a positive test when $1-\beta$ is large.

In particular, imagine a silly test with $1 - \beta = 0$. Then all significant results this test can produce are false-positives. So if you get a significant result from this silly test, it tells you nothing.

For further illustration, look at confidence intervals (CI). One may argue that larger sample size will make the CI more narrow and thus, if the test was significant for a smaller sample, it will also be significant for the larger sample. However, also the location of the CI can shift when we include more data in our sample, potentially making the result non-significant. It is also conceivable that the larger sample will have a much larger standard error and thus the CI will become wider in fact. One could say that a larger sample size gives the facts more opportunity to prove themselves.

There has been some interesting discussion lately about the interpretation of $p$-values, see, e.g.:

[1] Colquhoun, "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science, 2014

[2] Colquhoun, "The Reproducibility Of Research And The Misinterpretation Of P Values", 2017, http://www.biorxiv.org/content/early/2017/08/07/144337

[3] "What would Cohen say? A comment on $p < .005$", https://replicationindex.wordpress.com/2017/08/02/what-would-cohen-say-a-comment-on-p-005/

Concerning your particular result, I am not qualified to judge it. Using only your $p$-value and the classification from [2], it is between "weak evidence: worth another look" and "moderate evidence for a real effect".

Denote $p$ the $p$-value of your test (as a random variable) and fix some $\alpha$. Call a test result significant or positive when $p \leq \alpha$. We have $P(p \leq \alpha \,|\, H_0) \leq \alpha$. Moreover, let $\beta$ be such that $P(p > \alpha \,|\, H_1) \leq \beta$. Then $1-\beta$ is the power of the test.

Treating $H_0$ and $H_1$ as (complementary) events, Bayes' theorem gives: $$\frac{P(H_1 \, | \, p\leq\alpha)}{P(H_0 \,|\, p\leq\alpha)} = \frac{P(p\leq\alpha \,|\, H_1)}{P(p\leq\alpha \,|\, H_0)} \cdot \frac{P(H_1)}{P(H_0)} \geq \frac{1-\beta}{\alpha} \cdot \frac{P(H_1)}{P(H_0)}$$ This shows that the post odds for $H_1$ are a scaled version of the prior odds, with the strength of the scaling in favor for $H_1$ increasing with $1-\beta$. This means we learn more from a positive test when $1-\beta$ is large.

For further illustration, look at confidence intervals (CI). One may argue that larger sample size will make the CI more narrow and thus, if the test was significant for a smaller sample, it will also be significant for the larger sample. However, also the location of the CI can shift when we include more data in our sample, potentially making the result non-significant. It is also conceivable that the larger sample will have a much larger standard error and thus the CI will become wider in fact. One could say that a larger sample size gives the facts more opportunity to prove themselves.

There has been some interesting discussion lately about the interpretation of $p$-values, see, e.g.:

[1] Colquhoun, "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science, 2014

[2] Colquhoun, "The Reproducibility Of Research And The Misinterpretation Of P Values", 2017, http://www.biorxiv.org/content/early/2017/08/07/144337

[3] "What would Cohen say? A comment on $p < .005$", https://replicationindex.wordpress.com/2017/08/02/what-would-cohen-say-a-comment-on-p-005/

Concerning your particular result, I am not qualified to judge it. Using only your $p$-value and the classification from [2], it is between "weak evidence: worth another look" and "moderate evidence for a real effect".

Source Link
Lasse Kliemann
Lasse Kliemann
  • 159
  • 3

Denote $p$ the $p$-value of your test (as a random variable) and fix some $\alpha$. Call a test result significant or positive when $p \leq \alpha$. We have $P(p \leq \alpha \,|\, H_0) \leq \alpha$. Moreover, let $\beta$ be such that $P(p > \alpha \,|\, H_1) \leq \beta$. Then $1-\beta$ is the power of the test.

Treating $H_0$ and $H_1$ as (complementary) events, Bayes' theorem gives: $$\frac{P(H_1 \, | \, p\leq\alpha)}{P(H_0 \,|\, p\leq\alpha)} = \frac{P(p\leq\alpha \,|\, H_1)}{P(p\leq\alpha \,|\, H_0)} \cdot \frac{P(H_1)}{P(H_0)} \geq \frac{1-\beta}{\alpha} \cdot \frac{P(H_1)}{P(H_0)}$$ This shows that the post odds for $H_1$ are a scaled version of the prior odds, with the strength of the scaling in favor for $H_1$ increasing with $1-\beta$. This means we learn more from a positive test when $1-\beta$ is large.

In particular, imagine a silly test with $1 - \beta = 0$. Then all significant results this test can produce are false-positives. So if you get a significant result from this silly test, it tells you nothing.

For further illustration, look at confidence intervals (CI). One may argue that larger sample size will make the CI more narrow and thus, if the test was significant for a smaller sample, it will also be significant for the larger sample. However, also the location of the CI can shift when we include more data in our sample, potentially making the result non-significant. It is also conceivable that the larger sample will have a much larger standard error and thus the CI will become wider in fact. One could say that a larger sample size gives the facts more opportunity to prove themselves.

There has been some interesting discussion lately about the interpretation of $p$-values, see, e.g.:

[1] Colquhoun, "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science, 2014

[2] Colquhoun, "The Reproducibility Of Research And The Misinterpretation Of P Values", 2017, http://www.biorxiv.org/content/early/2017/08/07/144337

[3] "What would Cohen say? A comment on $p < .005$", https://replicationindex.wordpress.com/2017/08/02/what-would-cohen-say-a-comment-on-p-005/

Concerning your particular result, I am not qualified to judge it. Using only your $p$-value and the classification from [2], it is between "weak evidence: worth another look" and "moderate evidence for a real effect".