Repeating many ANOVA tests with a large number of factors I have a large number of factors that categorize my numerical observations in different ways and no preconception which if any of these factors may be significantly explanatory, but I have a p-value threshold that I'm obliged to observe. I want to perform an analysis of variance for each factor and then select those factors that have significant p-values according to my threshold. 
Because I'm running so many ANOVA tests, I'm worried about the possibility of a false discovery. Is this concern legitimate? And if so, what techniques are there for adjusting my p-value threshold to take account of the fact that I'm performing so many tests?
 A: You also seem a little unclear (possibly) on which tests you are conducting. You would generally conduct a single omnibus ANOVA to answer the question Do any of the samples in each of the levels of my factor variable have sample means that are drawn from a population with a different mean than at least one other factor? If you reject H$_{0}$ for ANOVA, you would then proceed to multiple pair-wise comparisons, and indeed take a false discovery rate approach (or family-wise error rate approach, if that's your druthers) to correcting the Type-I error probabilities of multiple t tests.
The different adjustment methods in order of least to most powerful (assuming you have made $m$ pair-wise comparisons):
Family-wise Error Rate Methods
Bonferroni: Compare $p$-values to  $\frac{\alpha}{m}$.
Šidák adjustment: Compare your $p$-values to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m}}$.
Holm's step-wise procedure:


*

*order your $p$-values from smallest to largest.

*Compare the smallest $p$ value to $\frac{\frac{\alpha}{2}}{m}$.

*Compare the second smallest $p$ value to $\frac{\frac{\alpha}{2}}{m-1}$.

*Compare the $i^{\text{th}}$ $p$-value to $\frac{\frac{\alpha}{2}}{m-i+1}$.

*For all tests following the first test for which you fail to reject H$_{0}$ you will also fail to reject H$_{0}$.


Holm-Šidák step-wise procedure:


*

*order your $p$-values from smallest to largest.

*Compare the smallest $p$ value to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m}}$.

*Compare the second smallest $p$ value to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m-1}}$.

*Compare the $i^{\text{th}}$ $p$-value to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m-i+1}}$.

*For all tests following the first test for which you fail to reject H$_{0}$ you will also fail to reject H$_{0}$.


False Discovery Rate
Benjamini & Hochberg's step-wise procedure: (appropriate when tests are independent, or when any dependency between tests is positive)


*

*order your $p$-values from largest to smallest.

*Compare the smallest $p$ value to $\frac{\alpha}{2}$.

*Compare the second smallest $p$ value to $\frac{m-1}{m}\times\frac{\alpha}{2}$.

*Compare the $i^{\text{th}}$ $p$-value to $\frac{m-i+1}{m}\times\frac{\alpha}{2}$.

*For all tests following the first test for which you reject H$_{0}$ we will also reject the null hypothesis.


Benjamini & Yekutieli's step-wise procedure: (appropriate when tests are dependent)


*

*order your $p$-values from largest to smallest.

*Compare the smallest $p$ value to $\frac{1}{C}\times\frac{\alpha}{2}$.

*Compare the second smallest $p$ value to $\frac{m-1}{mC}\times\frac{\alpha}{2}$.

*Compare the $i^{\text{th}}$ $p$-value to $\frac{m-i+1}{mC}\times\frac{\alpha}{2}$.

*For all tests following the first test for which you reject H$_{0}$ we will also reject the null hypothesis.


Where the constant $C = \sum_{i=1}^{m}{\frac{1}{i}}$
A: If I understand your question correctly, you have a multiple comparison issue. As a quick example of this issue, if you have a critical $\alpha$ value of 0.05 and you were to perform 100 hypothesis tests then we would expect 5 of the tests to come up significant even if all null hypothesizes  are actually true.
There are several ways to correct for this. The simplest (and most conservative) being the Bonferronni procedure where you divide the critical $\alpha$ value by the number of tests you are performing.  
