5
$\begingroup$

I have:

Condition #1 : 5 independent objects (1 object = a time series)
Condition #2 : 5 independent objects

My GOAL: test H0 that using condition #1 gives different results from using condition #2

I have a pairwise test statistics to test if one pair of objects are different. In total, I can calculate 5! pairwise comparisons => 120 p-values

Now I use Benjamini-Hochberg procedure to calculate adjusted p-values in R:

 p.adjust(pvalues, method="BH") 

(I can use Benjamini and Yekutieli instead for dependence, but lets skip this for now)

As far as I understand, this gives me a set of q-values, that is

"the adjusted p-value of an individual hypothesis is the lowest level of FDR for which the hypothesis is first included in the set of rejected hypotheses." (Reiner et.al, Bioinformatics, 2003).

Now I have 120 adjusted p.values, and I reject the null for all cases when p < 0.05. This would control the false discovery rate at 5%.

Can I say that "condition #1 gives different results from using condition #2" if number of rejected nulls is more than 120*5% = 6? In other words, since I expect 5% of false positives, can I say that the results are significant if I see more than 5% positives?

$\endgroup$
  • $\begingroup$ Actually, not. It would be "approximately" true if you would use FWER methods such as Bonferroni correction (you can have more than 5 FP results with low p-values just by chance). Now you use FDR, and it controls proportion of false positives, so it is more difficult to say if your results are significant or not. For real significance estimation is always better to have some prior knowledge of expected effect size. $\endgroup$ – German Demidov Feb 3 '16 at 16:36
3
$\begingroup$

Firstly we are going to formalize the hypotheses that you are testing

$$H_0 = \{ \text{using condition #1 gives the same results as using condition #2} \}$$

versus

$$H_1 = \{ \text{using condition #1 gives different results than using condition #2} \}$$

(I corrected your $H_0$, I think you want to formulate it like this)

Then you can formulate your hypothesis in terms of smaller hypothesis

$$H_0 = \cap_{i,j \in \{1,2,3,4,5\}} \{ \text{time series $i$ from condition 1 and time series $j$}$$

$$\quad \quad \quad \quad \text{ from condition 2 are comparable}\}$$

Is that correct?

Further, in this formulation there are only 5*5=25 smaller hypotheses to test, not 120...

With respect to our question,I would say no because significance implies that you are controling the probability for an type 1 error. Instead you are controling the FDR which is another concept.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.