12
$\begingroup$

Background: I read one article where authors report Pearson correlation 0.754 from sample size 878. Resulting p-value for correlation test is "two star" significant (i.e. p < 0.01). However, I think that with such a large sample size, corresponding p-value should be less than 0.001 (i.e. three star significant).

  • Can p-values for this test be computed just from Pearson correlation coefficient and sample size?
  • If yes, how can this be done in R?
Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
14
$\begingroup$

Yes, it can be done, if you use Fisher's R-to-z transformation. Other methods (e.g. bootstrap) can have some advantages but require the original data. In R (r is the sample correlation coefficient, n is the number of observations):

z <- 0.5 * log((1+r)/(1-r))
zse <- 1/sqrt(n-3)
min(pnorm(z, sd=zse), pnorm(z, lower.tail=F, sd=zse))*2

See also this post on my blog.

That said, whether it is .01 or .001 doesn't matter that much. As you said, this is mostly a function of sample size and you already know that the sample size is large. The logical conclusion is that you probably don't even need a test at all (especially not a test of the so-called ‘nil’ hypothesis that the correlation is 0). With N = 878, you can be quite confident in the precision of the estimate and focus on interpreting it directly (i.e. is .75 large in your field?).

Formally however, when you do a statistical test in the Neyman-Pearson framework, you need to specify the error level in advance. So, if the results of the test really matter and the study was planned with .01 as the threshold, it only makes sense to report p < .01 and you should not opportunistically make it p < .001 based on the obtained p value. This type of undisclosed flexibility is even one of the main reasons behind criticism of little stars and more generally of the way null-hypothesis significance testing is practiced in social science.

See also Meehl, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46 (4), 806-834. (The title contains a reference to these “stars” but the content is a much broader discussion of the role of significance testing.)

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ I would probably advise them to give up the little stars, even if the results are correct but I see your point. $\endgroup$
    – Gala
    Jun 6, 2013 at 6:50
  • 1
    $\begingroup$ I edited my answer to add a remark about this problem. Note that 0.001 < 0.01 so the authors are formally “correct” in any case, it's more a matter of what the way the results are reported implies. I would think that, unlike an outright error that a reviewer should of course correct, this issue should be left to the authors to decide. $\endgroup$
    – Gala
    Jun 6, 2013 at 7:02
  • 1
    $\begingroup$ You are right, but so far I have never seen reporting p < 0.01 if p is actually less than 0.001 (without saying that confidence level for article is 0.01). Moreover, in article that I speak about, authors report 30 correlation tests based on sample sizes ranging from 837 to 886 with correlations ranging from 0.145 to 0.754 and all are reported as two star significant. $\endgroup$
    – sitems
    Jun 6, 2013 at 7:12
  • 1
    $\begingroup$ I have a problem to post my code here, but I run simulations and p-value from your code is not the same as p-value from cor.test. $\endgroup$
    – sitems
    Jun 6, 2013 at 8:14
  • 4
    $\begingroup$ I wrote a tutorial review of the use of Fisher's z for correlations accessible at stata-journal.com/sjpdf.html?articlenum=pr0041 I'd recommend more use of confidence intervals and calculate 0.724, 0.781 as 95% limits. I'd recommend even more looking at the data and working out a regression. $\endgroup$
    – Nick Cox
    Jun 6, 2013 at 12:47
2
$\begingroup$

you use Fisher's R-to-z transformation.

There's an alternative statistic:

abs(r)*sqrt((n-2)/(1-r^2)) ~ t.dist(d.f.=n-2)

that has t-distribution with n-2 degrees of freedom. Which is how this works for example: http://www.danielsoper.com/statcalc3/calc.aspx?id=44

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.