Do I need to correct for multiple hypotheses? p-values all go to one

Question

I have two groups, C and A, where C is control. In both groups there are about 3,000 proteins. For each protein we acquire six measurements in both C and A (these are mass spectrometry intensities by the way). These measurements are paired.

I want to ask: "Are any of the proteins significantly different between C and A". I have performed 3,000 Wilcoxon paired tests and found some p-values < 0.05.

1. Is it correct to adjust for multiple hypotheses testing?

2. After I do correct all the p-values go to one, so nothing is significant. How should I interpret/present this?

3. Is it acceptable to rank the proteins according to the unadjusted p-values in order to determine which are most "interesting" and worth investigating further.

Answer 1

1. /2. Control over the probability of making even a single Type I error when all 3000 null hypotheses are true is too much to ask for. Controlling the expected proportion of Type I errors among those hypotheses you reject—the false discovery rate—is a more reasonable approach. See Benjamini & Hochberg (1995), "Controlling the false discovery rate: a practical and powerful approach to multiple testing". JRSS B, 57, 1.

2. Ranking by p-value is fine, but there'll be a rank order even in the absence of true differences between groups. Recall the distribution of p-values is uniform under the global null hypothesis, so examining if/how the empirical distribution deviates from uniformity should be useful.

Answer 2

1. That depends. If your question is really "are any different and which" you should in fact adjust for family-wise error rate. If your question is only "are there any differences, no matter where", you can try some high dimensional global test and don't need multiplicity adjustment. If your question is only "which are different" you can correct for false discovery rate. This is less strict. FDR just makes sure that you don't find too many proteins you should not care about if you knew truth.

2. You didn't find anything. Difference too small, sample size not well, no effect at all, you can never tell.

3. This is frequently done. In this case, the p-values are used as descriptive statistics. If you don't have missing data, this is (usually, as you might have used t-tests) equivalent to compare the standardized differences. As this procedure is embedded in your further investigation, you can do this. There the question is only "which protein is most likely to be different". This is no more a matter of rigorous inferential statistics, rather a Bayesian decision theory to make your research efficient.

By the way you can plan to order the p-values for using the Bonferroni-Holm procedure. This is more powerful than ordinary Bonferroni.

Answer 3

you have to find a different approach. if you correct via bonferroni method and chose family-wise alpha = .05 then the significance level for each test is 0,0000016. this means, it's extremely hard to get a significant result. unadjusted p-values mean nothing in this case as you have a chance for each of your 3000 tests to be a false positive of 5%, thus you would expect at least 150 alpha errors. try rearranging your data in order that you have to do fewer tests. the way you try to analyze it, imho, makes no sense.