Condition #1 : 5 independent objects (1 object = a time series)
Condition #2 : 5 independent objects
My GOAL: test H0 that using condition #1 gives different results from using condition #2
I have a pairwise test statistics to test if one pair of objects are different. In total, I can calculate 5! pairwise comparisons => 120 p-values
Now I use Benjamini-Hochberg procedure to calculate adjusted p-values in R:
(I can use Benjamini and Yekutieli instead for dependence, but lets skip this for now)
As far as I understand, this gives me a set of q-values, that is
"the adjusted p-value of an individual hypothesis is the lowest level of FDR for which the hypothesis is first included in the set of rejected hypotheses." (Reiner et.al, Bioinformatics, 2003).
Now I have 120 adjusted p.values, and I reject the null for all cases when p < 0.05. This would control the false discovery rate at 5%.
Can I say that "condition #1 gives different results from using condition #2" if number of rejected nulls is more than 120*5% = 6? In other words, since I expect 5% of false positives, can I say that the results are significant if I see more than 5% positives?