1
$\begingroup$

Two-tailed test on correlations:

$r = 0.04, n = 300, p = 0.50$ (not significant)

versus

$r= 0.18, n = 168, P = 0.02$ (significant)

I do not understand why there is such a huge difference in significance here.

$\endgroup$

1 Answer 1

3
$\begingroup$

Recall the definition of the p-value:

The p-value is the probability of obtaining a value of your test statistic as extreme as the one observed or more under the null hypothesis.

So: assuming the null hypothesis of no correlation, the p-value answers the question of how likely is it to get a correlation as large as you did or larger. (Or its negative, assuming a two-sided test.)

  • If you have 300 pairs of uncorrelated (that's your null hypothesis!) data points, you will pretty often get a correlation of $r\geq 0.04$ or $r\leq -0.04$, simply through random fluctuations.
  • However, if you have 168 uncorrelated pairs of data points, you will very rarely observe a correlation of $r\geq 0.18$ or $r\leq -0.18$. This correlation is statistically significant at the $\alpha=0.05$ level. You will deduce that the null hypothesis can be rejected.
$\endgroup$
9
  • $\begingroup$ What does it mean if we reject the null hypothesis? That there is some correlation present? $\endgroup$
    – AJJ
    Commented May 6, 2015 at 15:19
  • 1
    $\begingroup$ It just means what I wrote: if the null hypothesis were true, the observed test statistics would be very unlikely. We usually deduce that the null hypothesis is not true, i.e., that there is some correlation in the underlying data generation process. This is related to the mathematical proof by contradiction. Null hypothesis significance testing is not universally accepted, read the "criticism" section in the Wikipedia article. $\endgroup$ Commented May 6, 2015 at 15:23
  • $\begingroup$ So is p the probability that the results you obtained were the result of pure chance and not any kind of underlying relationship? $\endgroup$
    – AJJ
    Commented May 6, 2015 at 15:26
  • 2
    $\begingroup$ No. That is an extremely common misconception about p-values. It ignores base rates. I very much recommend that you read about criticisms of NHST, starting, e.g., with the Wikipedia articles linked above. $\endgroup$ Commented May 6, 2015 at 15:29
  • 1
    $\begingroup$ I know this is hard to wrap one's head around. See it this way: there is no probability that an underlying relationship exists or not. It either exists or not. What we can assign probabilities to are outcomes of random variables. This article is very good. $\endgroup$ Commented May 6, 2015 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.