Alternative to multiple Fisher's Exact tests

Question

I have count data for a pair of test and control populations of samples for a variable that takes a value of either A or B for a sample:

            A   |   B
         -------+-------
Control     7   |   1
Test        3   |   5

The hypothesis is that the test condition stimulates the conversion of the observed parameter from a state with a value of A to a state with a value of B, and I can use the Fisher Exact test on the data to accept or reject this hypothesis.

Now suppose we have three such pairs of test and control populations, with varying numbers of observations carried out for each population for the test or control condition.

                1                2                 3

            A   |   B        A   |   B         A   |   B
         -------+------   -------+-------   -------+-------
Control     7   |   1       70   |  10         7   |   1
Test        3   |   5        3   |   5       300   | 500

We can conduct the Exact test thrice, once for each pair, and obtain three P values, which we somehow assess together (majority rule?) to come to a conclusion.

But, is there some other statistical test that can be used, perhaps one that yields a single P value, or is more robust?

[Adding information about the actual experiment]

RNA molecules produced from a certain gene in the cells of the human body get edited because of exposure to the test condition, resulting in the conversion of a specific nucleotide base of the RNA from being an adenine (A; equivalent to the hypothetical state A above) to a guanine (G; equivalent to the hypothetical state B above). The editing is assayed by sequencing the RNA molecules, with varying number of molecules being sequenced for the control and test conditions. The three tables allude to data from three humans.

Answer 1

There are many ways to combine the information from the individual tests. Some examples follow. Where it makes sense to use ones near the top of the list, I'd lean toward those rather than the two at the end:

(a) If in the three situations, both test and control are believed to be independent draws from the same population of values (a 'test' population with constant proportion, and a control 'population' with its won constant proportion - just different sized samples being drawn in each case), then you can simply combine the data tables and test that. Point and interval estimates based on the combined data reflect the common population values.

(b) even when you don't assume a constant control proportion and a constant test proportion (as in (a) above), under the null the difference in proportions should still be zero. You can estimate the difference in proportion for each case and add the estimated proportions and add the variances of the estimates to construct a single statistic. If the difference in proportions were constant, you could get point and interval estimates for it, but the test still works as a test even when you don't assume a constant difference in proportion -- it will be sensitive to a tendency of the differences to be in the same direction. It would usually be reasonable to use a normal approximation for this test statistic, but you might also look at simulating distributions under the null.

(c) (again) in the case where the test and control proportions are not assumed to be constant across the three experiments under the alternative, you could still construct a statistic that combines information from the tables in other ways. One example would be to assume it's not the difference in proportions that's constant under the alternative, but that the log-odds is constant; you could then combine estimates of the log-odds (such as by forming weighted averages of them) and use that as an overall test statistic.

(d) You could combined (by addition) chi-square values for the individual tables; the chi-square approximation should be better in the combined case, though again it should be possible to construct simulated null distributions.

(e) if the tests are independent, you can use the Fisher procedure (see also here), which is effectively to multiply p-values, as one would tend to with independent probabilities (though by working on the log-scale, it's easier to compute the distribution).

If the nulls are true, the p-values have a uniform distribution. The $-2 \ln p_i$ should be exponentially distributed with mean $2$ (i.e. $\chi^2_2$) and adding those will give something that under the null should be $\chi^2_6$. If the combined result is unusually large for a $\chi^2_6$, you'd reject the null that the p-values were drawn from a uniform distribution in favor of the alternative that they tended to be smaller. In this particular case we have the slight problem that - even under the null - the p-values are discrete, so if the numbers are very small you might want to consider simulation under the null here as well.

(f) you could even add p-values. If the common nulls are true, the p-values (again) should be uniform; the sum of the p-values should have the distribution of a sum of uniforms; again this sum can be tested (in this case you test whether the sum of the p-values is too small to have come from a sum-of-uniforms), though again the discreteness may be an issue in some cases.

Where it's reasonable (from your prior knowledge of the situation) to make some assumptions (such as constant proportions, constant differences of proportions, constant log odds, or whatever) you should probably do so; this is usually more meaningful than say falling back on case (e), even though it's still a perfectly valid thing to do.

Answer 2

The following paper proposes a new, "inverted beta binomial" test to address this problem: a paired sample test for count data in a framework of statistical combination of multiple contingency tables.

Pham TZ and Jimenez CR. An accurate paired sample test for count data. Bioinformatics (2012) 28 (18): i596-i602. http://bioinformatics.oxfordjournals.org/content/28/18/i596.full

The paper notes three other statistical tests for the problem:

(1) Cochran–Mantel–Haenszel

(2) DerSimonian–Laird

(3) edgeR