Analysis of the Benjamini-Hochberg adjusted p-values

Question

I have:

Condition #1 : 5 independent objects (1 object = a time series)
Condition #2 : 5 independent objects

My GOAL: test H0 that using condition #1 gives different results from using condition #2

I have a pairwise test statistics to test if one pair of objects are different. In total, I can calculate 5! pairwise comparisons => 120 p-values

Now I use Benjamini-Hochberg procedure to calculate adjusted p-values in R:

 p.adjust(pvalues, method="BH") 

(I can use Benjamini and Yekutieli instead for dependence, but lets skip this for now)

As far as I understand, this gives me a set of q-values, that is

"the adjusted p-value of an individual hypothesis is the lowest level of FDR for which the hypothesis is first included in the set of rejected hypotheses." (Reiner et.al, Bioinformatics, 2003).

Now I have 120 adjusted p.values, and I reject the null for all cases when p < 0.05. This would control the false discovery rate at 5%.

Can I say that "condition #1 gives different results from using condition #2" if number of rejected nulls is more than 120*5% = 6? In other words, since I expect 5% of false positives, can I say that the results are significant if I see more than 5% positives?

Answer 1

Firstly we are going to formalize the hypotheses that you are testing

$$H_0 = \{ \text{using condition #1 gives the same results as using condition #2} \}$$

versus

$$H_1 = \{ \text{using condition #1 gives different results than using condition #2} \}$$

(I corrected your $H_0$, I think you want to formulate it like this)

Then you can formulate your hypothesis in terms of smaller hypothesis

$$H_0 = \cap_{i,j \in \{1,2,3,4,5\}} \{ \text{time series $i$ from condition 1 and time series $j$}$$

$$\quad \quad \quad \quad \text{ from condition 2 are comparable}\}$$

Is that correct?

Further, in this formulation there are only 5*5=25 smaller hypotheses to test, not 120...

With respect to our question,I would say no because significance implies that you are controling the probability for an type 1 error. Instead you are controling the FDR which is another concept.