How to implement multiple comparison correction after Wilcoxon rank-sum test with relativelty small sample sizes and many comparisons

Question

I am comparing 2 groups, performing a total of 180 comparisons. When doing these 180 pair-wise comparisons using the Wilcoxon rank-sum test more than 40 show a significant (p<0.05) difference between the groups (in all cases one group has a higher median value compared to the other group). The group sizes are relatively small: 20 and 25.

When using standard multiple comparison corrections (Bonferroni, Benjamini-Hochberg) all of the significant differences disappear. As the expected false discovery rate (assuming all null hypothesis are true) would be 180*0.05 = 9, I think these methods are very conservative.

Therefore, I have been looking into alternatives that might be suitable for many comparisons and relatively small sample sizes, and have a few options below.

1. Not correct for multiple comparison, as the chance of having so many positive findings by chance is less than 5%, a justification made in (1), where they state: "A correction for multiple comparisons was not necessary, because the number of channels with P-values below 0.05 ranged from 13 and 38 and the likelihood of having this many channels out of 150 by chance is less than 2% (cf. binomial distribution)."

2. A tmax permutation test. For this I am not sure what statistic would be most suitable, but I have attempted to use t-statistic, Welch's t-statistic and also O'Brian's test statistics (2) (I have not managed to implement the adjusted test). Almost all of the differences do disappear in these cases too.

I was wondering if there are any multiple comparison corrections that might be suitable for me that I have somehow missed? And any advice on how to proceed would be appreciated.

(1) Montez, T., Poil, S. S., Jones, B. F., Manshanden, I., Verbunt, J. P., van Dijk, B. W., Brussaard, A. B., van Ooyen, A., Stam, C. J., Scheltens, P., & Linkenkaer-Hansen, K. (2009). Altered temporal correlations in parietal alpha and prefrontal theta oscillations in early-stage Alzheimer disease. Proceedings of the National Academy of Sciences of the United States of America, 106(5), 1614–1619. https://doi.org/10.1073/pnas.0811699106

(2) Huang, P., Tilley, B. C., Woolson, R. F., & Lipsitz, S. (2005). Adjusting O'Brien's test to control type I error for the generalized nonparametric Behrens-Fisher problem. Biometrics, 61(2), 532–539. https://doi.org/10.1111/j.1541-0420.2005.00322.x

Answer 1

You are allowed to not make any p-value correction. You just need to understand and acknowledge that the chances of at least one of your reported significant results being a type-I error is relatively high.

It may be helpful to remember that the p-value threshold of 0.05 isn't magic.

This kind of approach makes sense, for example, in a screening study, with many treatments, to identify which treatments may warrant further study. Essentially because you are more concerned with type-II errors than type-I errors.