Example of strong correlation coefficient with a high p value

Question

I was wondering, is it possible to have a very strong correlation coefficient (say .9 or higher), with a high p value (say .25 or higher)?

Here's an example of a low correlation coefficient, with a high p value:

set.seed(10)
y <- rnorm(100)
x <- rnorm(100)+.1*y
cor.test(x,y)

cor = 0.03908927, p=0.6994

High correlation coefficient, low p value:

y <- rnorm(100)
x <- rnorm(100)+2*y
cor.test(x,y)

cor = 0.8807809, p=2.2e-16

Low correlation coefficient, low p value:

y <- rnorm(100000)
x <- rnorm(100000)+.1*y
cor.test(x,y)

cor = 0.1035018, p=2.2e-16

High correlation coefficient, high p value: ???

Answer 1

The Bottom Line

The sample correlation coefficient needed to reject the hypothesis that the true (Pearson) correlation coefficient is zero becomes small quite fast as the sample size increases. So, in general, no, you cannot simultaneously have a large (in magnitude) correlation coefficient and a simultaneously large $p$-value.

The Top Line (Details)

The test used for the Pearson correlation coefficient in the $R$ function cor.test is a very slightly modified version of the method I discuss below.

Suppose $(X_1,Y_1), (X_2,Y_2),\ldots,(X_n,Y_n)$ are iid bivariate normal random vectors with correlation $\rho$. We want to test the null hypothesis that $\rho = 0$ versus $\rho \neq 0$. Let $r$ be the sample correlation coefficient. Using standard linear-regression theory, it is not hard to show that the test statistic, $$ T = \frac{r \sqrt{n-2}}{\sqrt{(1-r^2)}} $$ has a $t_{n-2}$ distribution under the null hypothesis. For large $n$, the $t_{n-2}$ distribution approaches the standard normal. Hence $T^2$ is approximately chi-squared distributed with one degree of freedom. (Under the assumptions we've made, $T^2 \sim F_{1,n-2}$ in actuality, but the $\chi^2_1$ approximation makes clearer what is going on, I think.)

So, $$ \mathbb P\left(\frac{r^2}{1-r^2} (n-2) \geq q_{1-\alpha} \right) \approx \alpha \>, $$ where $q_{1-\alpha}$ is the $(1-\alpha)$ quantile of a chi-squared distribution with one degree of freedom.

Now, note that $r^2/(1-r^2)$ is increasing as $r^2$ increases. Rearranging the quantity in the probability statement, we have that for all $$ |r| \geq \frac{1}{\sqrt{1+(n-2)/q_{1-\alpha}}} $$ we'll get a rejection of the null hypothesis at level $\alpha$. Clearly the right-hand side decreases with $n$.

A plot

Here is a plot of the rejection region of $|r|$ as a function of the sample size. So, for example, when the sample size exceeds 100, the (absolute) correlation need only be about 0.2 to reject the null at the $\alpha = 0.05$ level.

A simulation

We can do a simple simulation to generate a pair of zero-mean vectors with an exact correlation coefficient. Below is the code. From this we can look at the output of cor.test.

k <- 100
n <- 4*k

# Correlation that gives an approximate p-value of 0.05
# Change 0.05 to some other desired p-value to get a different curve
pval <- 0.05
qval <- qchisq(pval,1,lower.tail=F)
rho  <- 1/sqrt(1+(n-2)/qval)

# Zero-mean orthogonal basis vectors
b1 <- rep(c(1,-1),n/2)
b2 <- rep(c(1,1,-1,-1),n/4)

# Construct x and y vectors with mean zero and an empirical
# correlation of *exactly* rho
x <- b1
y <- rho * b1 + sqrt(1-rho^2) * b2

# Do test
ctst <- cor.test(x,y)

As requested in the comments, here is the code to reproduce the plot, which can be run immediately following the code above (and uses some of the variables defined there).

png("cortest.png", height=600, width=600)
m  <- 3:1000
yy <- 1/sqrt(1+(m-2)/qval)
plot(m, yy, type="l", lwd=3, ylim=c(0,1),
     xlab="sample size", ylab="correlation")
polygon( c(m[1],m,rev(m)[1]), c(1,yy,1), col="lightblue2", border=NA)
lines(m,yy,lwd=2)
text(500, 0.5, "p < 0.05", cex=1.5 )
dev.off()

Answer 2

cor.test(c(1,2,3),c(1,2,2))

cor = 0.866, p = 0.333

Answer 3

A high estimate of the correlation coefficient with a high p-value could only occur with a very small sample size. I was about to provide an illustration, but Aaron has just done that!

Answer 4

I believe by the Fisher R-Z transform, the hyperbolic arctan of the sample correlation, under the null, is approximately normal with mean zero and standard error $1 / \sqrt{n-3}$. So to get, for example, a sample correlation $\hat{\rho} > 0$ with a fixed p-value, $p$, you would need $$p = 2 - 2 \Phi\left(\operatorname{atanh}(\hat{\rho})\sqrt{n-3}\right),$$ where $\Phi$ is the CDF of the standard normal, and you are performing a two-sided test for the null $H_0: \rho = 0$.

You can turn this into a function which gives the required $n$ for a fixed $\hat{\rho}$ and $p$. In R:

 #get n for sample correlation and p-value, 2-sided test of 0 correlation
 n.size <- function(rho.hat,p.val) {
   n <- 3 + ((qnorm(1 - 0.5 * p.val)) / atanh(rho.hat))^2
 }

Running this for $\hat{\rho} = 0.5$ and $p = 0.2$ gives:

print(n.size(0.5,0.2))

[1] 8.443062

So your sample size should be around 8. Playing around with this function should give you some idea of the relationship between $n, p$ and $\hat{\rho}$.

Answer 5

Yes. A p-value depends on the sample size, so a small sample can give this.

Say the true effect size was very small, and you draw a small sample. By luck, you get a few data points with very high correlation. The p-value will be high, as it should be. The correlation is high but it's not a very dependable result.

The sample correlation from R's cor() will tell you the best estimate of the correlation (given the sample). The p-value does NOT measure the strength of correlation. It measures how likely it could have arisen in case there actually was no effect, considering the size of the sample.

Another way to see this: If you have the same effect size, but get more samples, the p-value always goes to zero.

(If you want to more closely integrate the notions of estimated effect size and confidence about the estimate, it may be better to use confidence intervals; or, use Bayesian techniques.)