I am working with a set of genes for which I have both methylation values $\beta$ (continuous on unit interval) and gene expression $E$ (non-negative continuous) that I want to test for correlation $\hat\rho(log_2E,\beta)=R$. To filter out only the significant correlations I perform a permutation test where for each site I randomly permute $E$ a large number of times and calculate $R$. This gives me a distribution under the null hypothesis $p(R|H_0)$, i.e. when there is no dependence between $\beta$ and $E$, that I can compare $R$ against.

Now to my question,

I want to test both for positive and negative correlation. Do I make two tests per gene and use double multiple testing correction, where

$p_- = \int\limits_{-\inf}^Rp(x|H_0)dx$

$p_+ = \int\limits_{R}^\inf(x|H_0)dx$

or can and should I calculate a two-tailed p-value directly based on $|R|$ or such, and use normal multiple testing correction? I can only think of the following, but it feels wrong and has lousy power.

$p_2 = \int\limits_{-\inf}^{-|R|}p(x|H_0)dx + \int\limits_{|R|}^\inf(x|H_0)dx$

enter image description here

Edit: Updated the figure and added the missing x-axis label. The y-axis is the same for all panels.

  • 1
    $\begingroup$ What is your Null Hypothesis? That should determine how you derive the $p$ value from the permutation test. If the Null Hypothesis is that the correlation is 0 and that large positive and negative correlations are evidence against this Null, then you want the blue areas. $\endgroup$ – Reinstate Monica - G. Simpson Aug 10 '12 at 12:59
  • $\begingroup$ Yes, the null hypothesis would be that the correlation is 0. I just find it hard to understand why I get so much fewer hits than when combining the red and green tests. $\endgroup$ – Backlin Aug 10 '12 at 13:07
  • 2
    $\begingroup$ The green isn't really a test, it is the region that is consistent with the Null hypothesis in the one-sided test of 0 correlation where the alternative is that the correlation is negative. Actually, I wonder if your blues are too big? If you want a 95% two-sided test, the quantiles you want at the 0.25th and 0.975th such that the sum of the blue areas is 0.05 (5%). If a one-sided test, then all the evidence against the null is on one tail, in the two-sided evidence is in both tails. $\endgroup$ – Reinstate Monica - G. Simpson Aug 10 '12 at 13:17
  • 1
    $\begingroup$ You can't test for $R > 0$ here as otherwise you'd need to state something about how big $R$ is. Obviously the real $R$ is unknown otherwise you wouldn't be doing the test. The Null hypothesis is $R = 0$ and you are looking for evidence against this Null. I may have misunderstood what the plots show? Usually these plots would show the permutation distribution of $R$. $\endgroup$ – Reinstate Monica - G. Simpson Aug 10 '12 at 13:41
  • 1
    $\begingroup$ @andrea Okay that is true it is just a different way of looking at the same thing. $\endgroup$ – Michael R. Chernick Aug 10 '12 at 14:12

Here is an R example of permutation testing.

## dummy data
x <- runif(20)
y <- 0.5 * x
y <- y + rnorm(20)

## set up for the permutation, compute observed R
nullR <- numeric(length = 1000)
nullR[1] <- cor(x, y) ## obsered R in [1]
N <- length(x)

## permutation test
for(i in seq_len(999) + 1) {
    nullR[i] <- cor(x[sample(N)], y)

hist(nullR) ## histogram of R under H0

enter image description here

Now we can compute the permutation $p$ from the permutation distribution for the various tailed tests you consider:

> ## one side H1 R > 0
> sum(nullR >= nullR[1]) / length(nullR)
[1] 0.908
> ## one side H1 R < 0
> sum(nullR <= nullR[1]) / length(nullR)
[1] 0.093
> ## two sided
> sum(abs(nullR) >= abs(nullR[1])) / length(nullR)
[1] 0.177
  • $\begingroup$ I fail to see how this is any different than my examples, except that I used nullR to estimate the density $p(R|H_0)$ and then integrated rather than sum. For example isn't your first example (where $H_1: R>0$) more or less the same thing as my green case? $\endgroup$ – Backlin Aug 10 '12 at 18:40
  • $\begingroup$ Again, we might be talking at cross-purposes regarding the actual plots, but the way to look at this is via the blue plot. For R to be statistically significant at a given level, in the one tailed test with H1 R > 0 the observed R would need to be in the right hand blue area. For the H1 R < 0, the observed would have to be in the left hand blue area. For the two tailed test with H0 R = 0, H1 R!= 0, then the observed has to be in either of the blue areas. $\endgroup$ – Reinstate Monica - G. Simpson Aug 10 '12 at 18:52
  • $\begingroup$ @Backlin Your plots are confusing as the upper and lower panels show the regions for rejecting the Null in a one tailed test where H1 R < 0 (upper, red area) and rejecting the Null of a two tailed test where H1 != 0 (lower, blue areas). The green area in the middle plot is not the rejection area for the one-tailed test with H1 R > 0. I think this is what might be the cause of confusion between people here and yourself. In the one side H1 R > 0 we clearly have no evidence against H0 because the observed R lies in the green area. $\endgroup$ – Reinstate Monica - G. Simpson Aug 10 '12 at 18:55

Your formula for $p_2$ only works if you have distribution symmetric around $0$, which at least your illustration do not seem to have. A better estimate is given by $p_2(t) = {\rm min}(2p_{+}, 2p_{-})$. using your definitions of $p_{+}$ and $p_{-}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.