I am trying to get myself more savvy with literature on False Discovery Rates (FDR). At its original incarnation, FDR assumes that the tests being compared are independent. Several authors have published cases where the tests can have "clumpy dependencies" (tests are related to each other in "families", but not across families of tests), slight dependencies, etc., and still hold true. Most notably, Storey & Tibshirani ("Estimating the Positive False Discovery Rate Under Dependence...") give several ways the independence rule can be "bent" and how to do so.
However, I won't know a priori dependencies between tests, and dependencies may be different on the next set of test that I perform FDR upon. So, it seems to me that the easiest thing is to perform a principal components analysis (PCA) on all of the test data before calculating p-values and FDR.
What is the downside to performing PCA on the data set before performing FDR? The only down side that I can foresee is that I won't know which tests are found to be significant, since information is lost in performing that co-ordinate shift. But if my main concern is simply to determine whether 2 different levels across samples, for instance, are significantly different, I shouldn't need this information.
I haven't found any information combining the two methods, but it seems quite powerful to me. For one thing, PCA can shrink the data set immensely when looking at far more test parameters than observations (which is always the case in DNA microarrays, an example used often in the literature). But most importantly, I won't have to worry any more whether tests are overly dependent to apply FDR, since PCA assures orthogonality, obviously.
Being such a powerful combination (in my view), and seeing no literature on this subject, I "fear" I am missing something in combining these two methodologies.
PS Dependency obviously matters at some point: if all the tests were fully dependent, they provide the same information as 1 test, thus proper analysis should follow the standard 1-test methods. So this doesn't seem, to me, an extreme case or moot question.
Thanks to comments so far! Apologies for the imprecise nature of the question. Following is an example, although I plan to use this method for many different types of problems.
Hypothesis to test: using a new process, is there a marked improvement in process quality. My case is a chemical process, this is NOT gene expression.
Sample size: 144 different observations, split evenly into the two groups of "new process" and "POR" (process on record).
test size: there are potentially around 630 different tests that will be used to compare whether the new process is significantly different from POR.
Many of these tests are correlated with each other. Obviously due to the sample size being larger than the test size, even if the tests were not "correlated by nature", they become correlated due to the sparse data set.
BEFORE performing any t-tests or similar tests to generate a p-value, perform PCA upon the [144 x 630] matrix of test measurements.
Using the newly formed [144 x 144] matrix of PC's of test measurements, split them into "new process" and "POR".
Generate 144 test statistic p-values, comparing new process to POR
Perform FDR analysis on set of 144 statistics. I know this is far from the ~ 3000+ that are often used for FDR, but it still seems significant enough to use FDR instead of FWER, Bonferroni, or something similar.