Can we combine false discovery rate and principal components analysis?

Question

I am trying to get myself more savvy with literature on False Discovery Rates (FDR). At its original incarnation, FDR assumes that the tests being compared are independent. Several authors have published cases where the tests can have "clumpy dependencies" (tests are related to each other in "families", but not across families of tests), slight dependencies, etc., and still hold true. Most notably, Storey & Tibshirani ("Estimating the Positive False Discovery Rate Under Dependence...") give several ways the independence rule can be "bent" and how to do so.

However, I won't know a priori dependencies between tests, and dependencies may be different on the next set of test that I perform FDR upon. So, it seems to me that the easiest thing is to perform a principal components analysis (PCA) on all of the test data before calculating p-values and FDR.

What is the downside to performing PCA on the data set before performing FDR? The only down side that I can foresee is that I won't know which tests are found to be significant, since information is lost in performing that co-ordinate shift. But if my main concern is simply to determine whether 2 different levels across samples, for instance, are significantly different, I shouldn't need this information.

I haven't found any information combining the two methods, but it seems quite powerful to me. For one thing, PCA can shrink the data set immensely when looking at far more test parameters than observations (which is always the case in DNA microarrays, an example used often in the literature). But most importantly, I won't have to worry any more whether tests are overly dependent to apply FDR, since PCA assures orthogonality, obviously.

Being such a powerful combination (in my view), and seeing no literature on this subject, I "fear" I am missing something in combining these two methodologies.

Thanks!

PS Dependency obviously matters at some point: if all the tests were fully dependent, they provide the same information as 1 test, thus proper analysis should follow the standard 1-test methods. So this doesn't seem, to me, an extreme case or moot question.

---

Thanks to comments so far! Apologies for the imprecise nature of the question. Following is an example, although I plan to use this method for many different types of problems.

Hypothesis to test: using a new process, is there a marked improvement in process quality. My case is a chemical process, this is NOT gene expression.

Sample size: 144 different observations, split evenly into the two groups of "new process" and "POR" (process on record).

test size: there are potentially around 630 different tests that will be used to compare whether the new process is significantly different from POR.
Many of these tests are correlated with each other. Obviously due to the sample size being larger than the test size, even if the tests were not "correlated by nature", they become correlated due to the sparse data set.

Proposal:
BEFORE performing any t-tests or similar tests to generate a p-value, perform PCA upon the [144 x 630] matrix of test measurements.

Using the newly formed [144 x 144] matrix of PC's of test measurements, split them into "new process" and "POR".

Generate 144 test statistic p-values, comparing new process to POR

Perform FDR analysis on set of 144 statistics. I know this is far from the ~ 3000+ that are often used for FDR, but it still seems significant enough to use FDR instead of FWER, Bonferroni, or something similar.

Answer 1

The usual test scenario where you want to consider FDR is when you have a vector $(T_i)_{i=1,\ldots,n}$ of test statistics. The $i$'th test statistic could, for instance, be a test of differential expression between two cases (two levels in @Mike's terminology) for the $i$'th gene in a microarray experiment. The literature on FDR deals with choosing a threshold for the test statistics and providing an estimate of FDR for that threshold. How to do that correctly depends on the whether the test statistics are independent or dependent (and how they are dependent).

My understanding is that the question is based on a data matrix $X$ of dimension $m \times n$ of $m$ observations of an $n$-dimensional vector and that you want to use PCA to make a dimension reduction of $X$ to an $m \times n'$ matrix with orthogonal rows before you compute $n'$ test statistics. The test statistics could be tests of whether there is a statistically significant difference between two subgroups of the $m$ observations.

First, don't mix up the geometric orthogonality of the computed principal components with independence of the resulting test statistics. Although PCA "de-correlates" the columns above it does not imply statistical independence of the computed test statistics. However, there could be a point in reducing the $n$ to a (perhaps much smaller) $n'$. As @Mike remarks, if all the rows in the original data matrix are identical, say, there is really only one test, and if you do multiple testing corrections under the independence assumption you would be overly conservative. On the other hand, if only a small fraction of the rows represent differences between the two cases, then these differences might drown when you compute the principal components.

Whether the suggested method is a "powerful combination" or not depends upon the power of the resulting method. Will or will you not be able to detect the differences between the two cases across samples better by computing principal components first? The answer to this is a complicated function of the setup you consider and there will not be a simple answer. In the given example the use of PCA could equally well produce a set of directions where the differences have been smeared out as it could produce a set of directions that emphasize the differences. Given that you are only interested in detecting if there is a difference, it seems to me that you are really looking for a way to aggregate the data or the different test statistics. A simple idea, like taking the average, might work just as well.

Answer 2

(This is not yet an answer - but it may develop into one...)

What kind of measurements do you have?

If I had to compare spectra (just because that's the data I work with and you only told us what your data isn't, and 630 measurements per observation may very well be spectra - I'm quite willing to delete the answer if it doesn't fit the question), I'd probably do something along these lines:

• for a PCA, I'd rather center on the average before spectrum as that is a meaningful "base point"

• If I had to search for differences between the data sets before and after, I'd try to set up directly a supervised model, either a classification (LDA is related to PCA and PLS) or a regression (PLS, particularly if the inversion in the LDA is a problem) and then interprete that model.

• then I'd check how much the models change if I change the data set "a bit"- technically that's a cross validation or bootstrap. Just make sure you pick the correct "base unit" for splitting the data (do you need to leave out batches or days instead of single measurements?). You may need Procrustes-like roations for this. The variation between models give a very good idea what could be real and what is spurious.

• If the measurements have known physico-chemical meaning wrt. your process, checking this against the results / interpreting the models under this knowledge is a tremendous advantage over the usual settings of gene analysis. I'd consider "I'm not interested in which of them did change" throwing away much information that could help to judge what could be caused by the process change and what difference may just be random.

• With spectra I'd never consider univariate tests for each wavelength as something I'd seriously want to do (unless there are physico-chemical reasons to look for a difference at a particular wavelength): good spectra have a lot of correlation among neighbour wavelengths. This is a very different situation from "just" multiple testing (and if you have 630 different pH, T, p, ... measurements I imagine them being correlated about as much as spectra)