I want to test a sample correlation $r$ for significance, using p-values, that is

$H_0: \rho = 0, \; H_1: \rho \neq 0.$

I have understood that I can use Fisher's z-transform to calculate this by

$z_{obs}= \displaystyle\frac{\sqrt{n-3}}{2}\ln\left(\displaystyle\frac{1+r}{1-r}\right)$

and finding the p-value by

$p = 2P\left(Z>z_{obs}\right)$

using the standard normal distribution.

My question is: how large $n$ should be for this to be an appropriate transformation? Obviously, $n$ must be larger than 3. My textbook does not mention any restrictions, but on slide 29 of this presentation it says that $n$ must be larger than 10. For the data I will be considering, I will have something like $5 \leq n \leq 10$.

  • 2
    $\begingroup$ The Wikipedia page lists the standard error of $z_{obs}$ which is given by $1/\sqrt{N-3}$ where $N$ is the sample size. So you'll need at least 4 complete pairs. I'm not aware of any restrictions beyond that regarding sample size. $\endgroup$ – COOLSerdash Jun 28 '13 at 11:52
  • 8
    $\begingroup$ Not sure how much to trust a presentation from someone who can't spell their own university name. More seriously, beware all advice that implies that things are fine above a certain sample size and dire otherwise. It's a matter of approximation quality increasing smoothly with sample size and also depending on the distribution of the data. Simple advice is to be very cautious, plot everything and cross-check with bootstrapped confidence intervals. $\endgroup$ – Nick Cox Jun 28 '13 at 12:04
  • 1
    $\begingroup$ Slide 17 describes a t-test for the special case $\rho=0$. $\endgroup$ – whuber Jun 28 '13 at 14:07

For questions like these I would just run a simulation and see if the $p$-values behave as I expect them to. The $p$-value is the probability of randomly drawing a sample that deviates at least as much from the null-hypothesis as the data you observed if the null-hypothesis is true. So if we had many such samples, and one of them had a $p$-value of .04 then we would expect 4% of those samples to have a value less than .04. The same is true for all other possible $p$-values.

Below is a simulation in Stata. The graphs check whether the $p$-values measure what they are supposed to measure, that is, they shows how much the proportion of samples with $p$-values less than the nominal $p$-value deviates from the nominal $p$-value. As you can see that test is somewhat problematic with such small number of observations. Whether or not it is too problematic for your research is your judgement call.

clear all
set more off

program define sim, rclass
    tempname z se
    foreach i of numlist 5/10 20(10)50 {
        drop _all
        set obs `i'
        gen x = rnormal()
        gen y = rnormal()
        corr x y 
        scalar `z'  = atanh(r(rho))
        scalar `se' = 1/sqrt(r(N)-3)
        return scalar p`i' = 2*normal(-abs(`z'/`se'))

simulate p5 =r(p5)  p6 =r(p6)  p7  =r(p7)     ///
         p8 =r(p8)  p9 =r(p9)  p10 =r(p10)    ///
         p20=r(p20) p30=r(p30) p40 =r(p40)    ///
         p50=r(p50), reps(200000) nodots: sim 

simpplot p5 p6 p7 p8 p9 p10, name(small, replace) ///
    scheme(s2color) ylabel(,angle(horizontal)) 

enter image description here

simpplot p20 p30 p40 p50 , name(less_small, replace) ///
    scheme(s2color) ylabel(,angle(horizontal)) 

enter image description here

  • 1
    $\begingroup$ Try subtracting 2.5 instead of 3 from $n$ :-). $\endgroup$ – whuber Jun 28 '13 at 14:09

FWIW I see the recommendation $N\ge 10$ in Myers & Well (research design and statistical analyses, second edition, 2003, p. 492). The footnote states:

Strictly speaking, the $Z$ transformation is biased by an amount $r/(2(N-1))$: see Pearson and Hartley (1954, p. 29). This bias will generally be negligible unless $N$ is small and $\rho$ is large, and we ignore it here.

  • 3
    $\begingroup$ This does seem like it is an answer to me. $\endgroup$ – gung Apr 5 '16 at 14:42

Not sure whether a Fisher's $z$ transform is appropriate here. For $H_0: \rho=0$ (NB: null hypothesis is for population $\rho$, not sample $r$), the sampling distribution of the correlation coefficient is already symmetric, so no need to reduce skewness, which is what Fisher's $z$ aims to do, and you can use Student's $t$ approximation.

Assuming you mean $H_0: \rho = \rho_0 \not = 0$, then the skewness of that PDF will depend on the proposed value of $\rho_0$, so there would then be no general answer of how large $n$ should be. Also, minimum values of $n$ would depend on the significance level $\alpha$ that you are working toward. You did not state its value.

Nick's point is a fair one: the approximations and recommendations are always operating in some grey area.

If, then, your Fisher approximation is good (=symmetric) enough, I would use the bound $n\geq (t_{\alpha/2} s/\epsilon)^2$ applicable to $t$-distributions, where $s$ is the sample standard deviation. If it is close enough to normality, this becomes $n \geq (1.96 s/\epsilon)^2$.

  • 4
    $\begingroup$ I think this oversimplifies the "aim" of Fisher's $z$, which is partly a matter of purpose as well as mathematics. Skewness or not is only part of the picture; $z$ transforms a bounded distribution to an unbounded one, which is important for confidence intervals. In fact, I would argue that unless a null hypothesis of zero correlation is the scientific question too, the use of Fisher's $z$ for confidence intervals is much more fruitful than trying to get a P-value. $\endgroup$ – Nick Cox Jun 28 '13 at 13:35
  • 1
    $\begingroup$ I'm sorry, I am new to the Fisher's $z$-transform. Should I only use it if I want to test $H_0: \rho = \rho_0 \neq 0$? The reason for calculating P-values is that I want to use the Holm-Bonferroni method to control family-wise error rate when doing multiple comparisons. Should I rather calculate P-values from a Student's $t $ distribution? $\endgroup$ – Gunnhild Jun 28 '13 at 13:48
  • 3
    $\begingroup$ Question is the wrong way round, I think. Fisher's $z$ is a better method for confidence intervals and for inference generally. Most software, I guess, uses a $t$-based calculation for testing $\rho = 0$. If in doubt it could be really important to show whether using one method makes a difference for your data. So, if methods agree, there is no problem. $\endgroup$ – Nick Cox Jun 28 '13 at 13:56
  • 1
    $\begingroup$ You can read more about Fisher's $z$ transformation here: stata-journal.com/article.html?article=pr0041 $\endgroup$ – Maarten Buis Jun 28 '13 at 13:59
  • $\begingroup$ Ok, thank you @NickCox! @Lucozade, what is the $\epsilon$ in the bound on $n$ ? $\endgroup$ – Gunnhild Jun 28 '13 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.