One-sided or two-sided Wilcoxon rank-sum test depending on fold change

Question

I am looking at fold changes $\frac{B}{A}$ over many different genes and I want to test with a Wilcoxon rank-sum test if the samples significantly differ.

I want to have as much power as possible because I will perform FDR correction afterwards, so I want to use the one-sided test with more power. Is it legitimate to look at the fold change for each gene before deciding if I perform a one-sided test with alternative 'greater' or 'less'?

If I take the wrong assumption the p-value is very high and this interferes with my FDR correction afterwards.

Answer 1

A consequent version of your process would be to perform two one-sided tests per gene and then pick the smaller of the two p values. To correct for the corresponding inflation of the type I error associated with each gene, you would need to apply some form of multiple testing correction. Otherwise, the effective type I error per gene would be grossly underestimated.

Here a very small simulation in R, assuming there are no effects.

one_gene <- function(n = 30) {
  y1 <- rnorm(n)
  y2 <- rnorm(n)
  
  greater <- wilcox.test(y1, y2, alternative = "greater")$p.value
  less <- wilcox.test(y1, y2, alternative = "less")$p.value

  return(min(greater, less))
}

set.seed(10)
many_genes <- replicate(1000, one_gene())

mean(many_genes <= 0.05)  # 0.11
hist(many_genes)

Indeed: the probability for a type I error is about 10%, i.e. twice 5%. And the histogram of the p values is not uniform over [0, 1] but rather uniform over [0, 0.5]. As a consequence, you would need to apply e.g. Bonferroni's correction for multiple testing, but in this case the much, much more natural thing to do is:

Just test two-sided. It is the right thing to do in your case.