Repeating many ANOVA tests with a large number of factors

I have a large number of factors that categorize my numerical observations in different ways and no preconception which if any of these factors may be significantly explanatory, but I have a p-value threshold that I'm obliged to observe. I want to perform an analysis of variance for each factor and then select those factors that have significant p-values according to my threshold.

Because I'm running so many ANOVA tests, I'm worried about the possibility of a false discovery. Is this concern legitimate? And if so, what techniques are there for adjusting my p-value threshold to take account of the fact that I'm performing so many tests?

-
Different methods include - Benferroni correction (a conservative measure), FDR or permutation based p-value threshold calculation ... –  Ram Sharma Jun 30 at 17:31
@RamSharma And the Sidak adjustment, the Holm stepwise adjustment, the and the Holm-Sidak stepwise adjustment, the last being the most powerful of the methods listed between your and my FWER methods. :) –  Alexis Jun 30 at 17:36

You also seem a little unclear (possibly) on which tests you are conducting. You would generally conduct a single omnibus ANOVA to answer the question Do any of the samples in each of the levels of my factor variable have sample means that are drawn from a population with a different mean than at least one other factor? If you reject H$_{0}$ for ANOVA, you would then proceed to multiple pair-wise comparisons, and indeed take a false discovery rate approach (or family-wise error rate approach, if that's your druthers) to correcting the Type-I error probabilities of multiple t tests.

The different adjustment methods in order of least to most powerful (assuming you have made $m$ pair-wise comparisons):

Family-wise Error Rate Methods

Bonferroni: Compare $p$-values to $\frac{\alpha}{m}$.

Šidák adjustment: Compare your $p$-values to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m}}$.

Holm's step-wise procedure:

1. order your $p$-values from smallest to largest.
2. Compare the smallest $p$ value to $\frac{\frac{\alpha}{2}}{m}$.
3. Compare the second smallest $p$ value to $\frac{\frac{\alpha}{2}}{m-1}$.
4. Compare the $i^{\text{th}}$ $p$-value to $\frac{\frac{\alpha}{2}}{m-i+1}$.
5. For all tests following the first test for which you fail to reject H$_{0}$ you will also fail to reject H$_{0}$.

Holm-Šidák step-wise procedure:

1. order your $p$-values from smallest to largest.
2. Compare the smallest $p$ value to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m}}$.
3. Compare the second smallest $p$ value to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m-1}}$.
4. Compare the $i^{\text{th}}$ $p$-value to $1-\left(1-\frac{\alpha}{2}\right)^{\frac{1}{m-i+1}}$.
5. For all tests following the first test for which you fail to reject H$_{0}$ you will also fail to reject H$_{0}$.

False Discovery Rate

Benjamini & Hochberg's step-wise procedure:

1. order your $p$-values from largest to smallest.
2. Compare the smallest $p$ value to $\frac{\alpha}{2}$.
3. Compare the second smallest $p$ value to $\frac{m-1}{m}\times\frac{\alpha}{2}$.
4. Compare the $i^{\text{th}}$ $p$-value to $\frac{m-i+1}{m}\times\frac{\alpha}{2}$.
5. For all tests following the first test for which you reject H$_{0}$ we will also reject the null hypothesis.
-
Thanks. By factor, I mean a categorical variable with 3 or more levels. –  Matthew Plourde Jun 30 at 17:46
Yes, that's right. You still perform t tests not additional ANOVAs subsequent to rejecting the null hypothesis of a oneway ANOVA. –  Alexis Jun 30 at 19:39
OK, you are describing how to get to the pvalues of pair-wise combinations of the levels of a single factor. But I've got bunch of factors, and I'm trying to identify which one's are significant at the omnibus level. –  Matthew Plourde Jun 30 at 19:49
No, I am not. Here's the procedure step by step: (1) ANOVA (1a) Not reject? Done! No pairwise tests for difference are rejected at your level of $\alpha$. (1b) Reject: proceed to step 2; (2) Conduct pairwise tests. If you have $k$ factors, then you may perform up to $\frac{k\left(k-1\right)}{2}$ of them; let's call $m$ the number of pairwise tests. You perform $m$ unpaired t tests for differences in means in one factor level versus another; these t tests use pooled variance equal to the "within variance" from your $F$ statistic. (3) Adjust your $p$ values (e.g. using FDR). –  Alexis Jun 30 at 19:59

If I understand your question correctly, you have a multiple comparison issue. As a quick example of this issue, if you have a critical $\alpha$ value of 0.05 and you were to perform 100 hypothesis tests then we would expect 5 of the tests to come up significant even if all null hypothesizes are actually true.

There are several ways to correct for this. The simplest (and most conservative) being the Bonferronni procedure where you divide the critical $\alpha$ value by the number of tests you are performing.

-