# Two-sided permutation test vs. two one-sided

I am working with a set of genes for which I have both methylation values $\beta$ (continuous on unit interval) and gene expression $E$ (non-negative continuous) that I want to test for correlation $\hat\rho(log_2E,\beta)=R$. To filter out only the significant correlations I perform a permutation test where for each site I randomly permute $E$ a large number of times and calculate $R$. This gives me a distribution under the null hypothesis $p(R|H_0)$, i.e. when there is no dependence between $\beta$ and $E$, that I can compare $R$ against.

Now to my question,

I want to test both for positive and negative correlation. Do I make two tests per gene and use double multiple testing correction, where

$p_- = \int\limits_{-\inf}^Rp(x|H_0)dx$

$p_+ = \int\limits_{R}^\inf(x|H_0)dx$

or can and should I calculate a two-tailed p-value directly based on $|R|$ or such, and use normal multiple testing correction? I can only think of the following, but it feels wrong and has lousy power.

$p_2 = \int\limits_{-\inf}^{-|R|}p(x|H_0)dx + \int\limits_{|R|}^\inf(x|H_0)dx$

Edit: Updated the figure and added the missing x-axis label. The y-axis is the same for all panels.

-
What is your Null Hypothesis? That should determine how you derive the $p$ value from the permutation test. If the Null Hypothesis is that the correlation is 0 and that large positive and negative correlations are evidence against this Null, then you want the blue areas. – Gavin Simpson Aug 10 '12 at 12:59
Yes, the null hypothesis would be that the correlation is 0. I just find it hard to understand why I get so much fewer hits than when combining the red and green tests. – Backlin Aug 10 '12 at 13:07
The green isn't really a test, it is the region that is consistent with the Null hypothesis in the one-sided test of 0 correlation where the alternative is that the correlation is negative. Actually, I wonder if your blues are too big? If you want a 95% two-sided test, the quantiles you want at the 0.25th and 0.975th such that the sum of the blue areas is 0.05 (5%). If a one-sided test, then all the evidence against the null is on one tail, in the two-sided evidence is in both tails. – Gavin Simpson Aug 10 '12 at 13:17
You can't test for $R > 0$ here as otherwise you'd need to state something about how big $R$ is. Obviously the real $R$ is unknown otherwise you wouldn't be doing the test. The Null hypothesis is $R = 0$ and you are looking for evidence against this Null. I may have misunderstood what the plots show? Usually these plots would show the permutation distribution of $R$. – Gavin Simpson Aug 10 '12 at 13:41
@andrea Okay that is true it is just a different way of looking at the same thing. – Michael Chernick Aug 10 '12 at 14:12

Here is an R example of permutation testing.

## dummy data
set.seed(1)
x <- runif(20)
y <- 0.5 * x
y <- y + rnorm(20)

## set up for the permutation, compute observed R
nullR <- numeric(length = 1000)
nullR[1] <- cor(x, y) ## obsered R in [1]
N <- length(x)

## permutation test
for(i in seq_len(999) + 1) {
nullR[i] <- cor(x[sample(N)], y)
}

hist(nullR) ## histogram of R under H0


Now we can compute the permutation $p$ from the permutation distribution for the various tailed tests you consider:

> ## one side H1 R > 0
> sum(nullR >= nullR[1]) / length(nullR)
[1] 0.908
> ## one side H1 R < 0
> sum(nullR <= nullR[1]) / length(nullR)
[1] 0.093
> ## two sided
> sum(nullR >= abs(nullR[1])) / length(nullR)
[1] 0.084

-
 I fail to see how this is any different than my examples, except that I used nullR to estimate the density $p(R|H_0)$ and then integrated rather than sum. For example isn't your first example (where $H_1: R>0$) more or less the same thing as my green case? – Backlin Aug 10 '12 at 18:40 Again, we might be talking at cross-purposes regarding the actual plots, but the way to look at this is via the blue plot. For R to be statistically significant at a given level, in the one tailed test with H1 R > 0 the observed R would need to be in the right hand blue area. For the H1 R < 0, the observed would have to be in the left hand blue area. For the two tailed test with H0 R = 0, H1 R!= 0, then the observed has to be in either of the blue areas. – Gavin Simpson Aug 10 '12 at 18:52 @Backlin Your plots are confusing as the upper and lower panels show the regions for rejecting the Null in a one tailed test where H1 R < 0 (upper, red area) and rejecting the Null of a two tailed test where H1 != 0 (lower, blue areas). The green area in the middle plot is not the rejection area for the one-tailed test with H1 R > 0. I think this is what might be the cause of confusion between people here and yourself. In the one side H1 R > 0 we clearly have no evidence against H0 because the observed R lies in the green area. – Gavin Simpson Aug 10 '12 at 18:55