Comparing multiple contingency tables, independent data (migrating from math overflow, where no answers were posted)
suppose I have $K$ different methods for forecasting a binary random variable, which I test on independent sets of data, resulting in $K$ contingency tables of values $n_{ijk}$ for $i,j=1,2$ and $k=1,2,...,K$. How can I compare these methods based on the contingency tables?  The general case would be nice, but $K=2$ is also very interesting.
I can think of a few approaches:


*

*compute some statistic on each of the tables, and compare those random variables (I'm not sure if this is a standard problem or not),

*something like Goodman's improvement of Stouffer's method, but I cannot access this paper, and was hoping for something a little more recent (more likely to have the latest-greatest, plus computer simulations).


any ideas?
 A: I may be a little unclear about the question.  But here would be my solution computing some "statistic on each of the tables" and comparing those values.
If your contingency tables are like a binomial effect size display (BESD), with clear YES/NO predictions being provided by each of your K methods you'll have a number of tables like this...

                 Reality
             +    - 
Pred  +     70   30 100
Pred  -     30   70 100
            100 100 200

I believe you can find the difference between the success rates, e.g. 70/100 – 30/100 = 40/100 = .40, this value can be considered as being equivalent to an effect size r for each of your K methods.  As a proof of concept I've included equivalent R code...
x <- rep(c(1,0),each=100)
y <- c(rep(1,70),rep(0,30),rep(1,30),rep(0,70))
cor(x,y)

You can then compare them using Fisher's Z' transformation for r in the standard way, e.g. here.  To deal with situations where K is greater than 2 one may want to apply some familywise error correction to the Z' tests, but the exact one selected I leave open for another debate.  P.S. I might be remembering incorrectly, but I think you can find more details in Essentials of Behavioral Research: Methods and Data Analysis by Rosenthal & Rosnow, 2007, Ch 11
A: Your questions is a good one (given I understand correctly). I believe you have K, 2x2 tables which correspond K different methods (call Z) and your aim is to say .. method K_1, K_2 ... K_n (K_i belongs to {1,...,K}) have some association between prediction and truth and the remaining don't have a relation. If you think this is a correct interpretation, proceed ahead o/w ignore my answer.
The problem is similar to partial tables when we control for Z (ie condition on a particular method) and study the XY (truth and prediction) relationship at fixed levels of Z. Basically, two way cross-sectional slices of the three way contingency table cross classify X and Y at several values of Z (1 .. K for you). These cross sections are called Partial Tables. 
Associations for partial tables are called conditional associations, because they refer to the effect of X on Y conditional of fixing Z at some level. 
For reference see: Agresti's Categorical Data Analysis Book link. The example 2.3.2 (Death penalty example) is a special case and talks about 2x2x2 tables. Section 2.3.3 answers your question for the general scenario. Your question can be answered by testing if there is conditional independence.
This is accomplished using CMH (Cochran Mantel Haenszel) Test of Conditional independence. In R, this can be done using this function. CMH generalizations for IxJxK tables also exist. In your case under the null, the distribution of test statistic is hypergeometric and for greater details see Section 6.3 of the Agresti book.
Hopefully this answers your questions. Otherwise, I am sorry for making you read this far.
