Comment: heavily edited, both to clarify and to fix a mistake. First 7 comments below refer to the previous version.
I have an experiment where 3 raters go over the same data 2 times each, trying to make a diagnosis. The second of these two passes serves as the gold standard, and I want to compare the results of the first one against certain variable (info source, two possible values). Specifically, I am interested in the cases where the raters were not sure of their diagnosis.
So, I have a 2x2 table for each rater: counts of "unsure" cases becoming (or not) "sure" in the gold standard, vs. info source. The inter-rater agreement is very poor (being unsure is a very subjective thing; FWIW, when they are sure, they agree pretty well), but the data is obviously not independent (so I can't use Cochran-Mantel-Haenszel, can I?). So – is there any reasonable (and preferably reasonably standard) way to combine these three 2x2 tables to tell whether there is a dependence on the information source?