P-value correction/adjustment for multiple groups and high number of tests My dataset is 5175 rows by 16 columns. Each row is a biological pathway, each column is a sample (n=4/group), and each observation is an enrichment score. Due to the small sample size and non-normal distribution, I am choosing nonparametric tests.
My goal is to first run a Kruskal-Wallis test between the four groups for each biological pathway and then apply the appropriate post-hoc tests.
My questions are as follows:


*

*What is the appropriate post-hoc test following KW? I have heard both Dunn and Mann-Whitney can work but am not sure

*Do I need to apply an FDR correction to the Kruskal-Wallis test due to the fact that I will be running >5000 tests?

*For the post-hoc pairwise comparisons, how do I simultaneously adjust for multiple comparisons (i.e. 4 groups) as well as for the fact that I am running >5000 tests?


Example R code:
#run KW test for each row
data$KW.pvalue <- apply(data,1, function(x) {
kruskal.test(values~ind,
           data=stack(data.frame(cbind(
             groupA=as.numeric(x[1:4]),
             groupB=as.numeric(x[5:8]),
             groupC=as.numeric(x[9:12]),
             groupD=as.numeric(x[13:16]))
             )))$p.value})

 A: Are you familiar with the Bioconductor project? I suggest that you have a look at R software packages in that project, because they are specifically designed to address statistical bioinformatics problems such as yours. A lot of statistical methodology has been developed in this field and simply applying univariate tests like kruskal.test() row by row is hardly ever an optimal way to proceed. The Bioconductor support forum is very active and you would be able to get feedback about how to proceed.
You don't say where your enrichment scores are from. Have you perhaps output enrichment scores using ssGSEA from the Broad Institute?
You don't say what the pathways are. Are they perhaps gene sets from the Molecular Signatures Database?
The problems of small samples and of multiple testing over >5000 pathways are far, far more important to deal with than issues of posthoc tests or non-normality. One way to proceed would be to transform the enrichment scores to rough normality (it only needs to be rough) and then use the limma software package to do a differential enrichment analysis. (Note I'm the limma author.) limma conducts t-tests and F-tests and uses empirical Bayes methods to leverage information from all the rows at once. limma offers rich possibilities for FDR control over both multiple rows and multiple comparisons. It is also provides far better statistical power than a univariate test like Kruskal's can give.
Alternatively, you could use limma's pathway analysis functions directly. Limma functions like roast(), camera() or fry() can conduct differential pathway enrichment analyses directly from your raw data, without computing single-sample enrichment scores.
A: @Gordon_Smyth is correct, I regularly use his package for my works. I wouldn't apply KW test row-by-row in your case unless you have a small number of genes (but you have 5175 genes). I'd let the limma package work out the details.


*

*If you really want to do KW. Dunn's Test is fine. (Post-hoc tests after Kruskal-Wallis: Dunn's test or Bonferroni corrected Mann-Whitney tests?)

*Yes. Not applying any correlation for your 5175 genes is a statistical mistake.

*You don't really need to adjust for pairwise comparisons.

