0
$\begingroup$

I am looking at fold changes $\frac{B}{A}$ over many different genes and I want to test with a Wilcoxon rank-sum test if the samples significantly differ.

I want to have as much power as possible because I will perform FDR correction afterwards, so I want to use the one-sided test with more power. Is it legitimate to look at the fold change for each gene before deciding if I perform a one-sided test with alternative 'greater' or 'less'?

If I take the wrong assumption the p-value is very high and this interferes with my FDR correction afterwards.

$\endgroup$
3
  • $\begingroup$ No. This would badly invalidate the whole process. $\endgroup$
    – Michael M
    May 15, 2021 at 16:55
  • $\begingroup$ Could you explain why? Because if I want to test if one group is higher or lower I have to specify the alternative. $\endgroup$
    – samu
    May 15, 2021 at 17:10
  • $\begingroup$ A two sided test is the way to go. Your procedure has an alpha level of 2 alpha (taking the smaller of the two p values), so you won't gain anything. $\endgroup$
    – Michael M
    May 15, 2021 at 19:06

1 Answer 1

1
$\begingroup$

A consequent version of your process would be to perform two one-sided tests per gene and then pick the smaller of the two p values. To correct for the corresponding inflation of the type I error associated with each gene, you would need to apply some form of multiple testing correction. Otherwise, the effective type I error per gene would be grossly underestimated.

Here a very small simulation in R, assuming there are no effects.


one_gene <- function(n = 30) {
  y1 <- rnorm(n)
  y2 <- rnorm(n)
  
  greater <- wilcox.test(y1, y2, alternative = "greater")$p.value
  less <- wilcox.test(y1, y2, alternative = "less")$p.value

  return(min(greater, less))
}

set.seed(10)
many_genes <- replicate(1000, one_gene())

mean(many_genes <= 0.05)  # 0.11
hist(many_genes)

enter image description here

Indeed: the probability for a type I error is about 10%, i.e. twice 5%. And the histogram of the p values is not uniform over [0, 1] but rather uniform over [0, 0.5]. As a consequence, you would need to apply e.g. Bonferroni's correction for multiple testing, but in this case the much, much more natural thing to do is:

Just test two-sided. It is the right thing to do in your case.

$\endgroup$
1
  • $\begingroup$ I see, that absolutely makes sense, I also thought it was quite a hack. Thank you for the clear and nice explanation! But then I also correct for the two-sided tests using, for example, the Benjamini-Hochberg procedure? $\endgroup$
    – samu
    May 16, 2021 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.