Comment: heavily edited, both to clarify and to fix a mistake. First 7 comments below refer to the previous version.

I have an experiment where 3 raters go over the same data 2 times each, trying to make a diagnosis. The second of these two passes serves as the gold standard, and I want to compare the results of the first one against certain variable (info source, two possible values). Specifically, I am interested in the cases where the raters were not sure of their diagnosis.

So, I have a 2x2 table for each rater: counts of "unsure" cases becoming (or not) "sure" in the gold standard, vs. info source. The inter-rater agreement is very poor (being unsure is a very subjective thing; FWIW, when they are sure, they agree pretty well), but the data is obviously not independent (so I can't use Cochran-Mantel-Haenszel, can I?). So – is there any reasonable (and preferably reasonably standard) way to combine these three 2x2 tables to tell whether there is a dependence on the information source?

  • $\begingroup$ Some additional clarity would be helpful. In paragraph 1, '3 raters 3 times each' sounds like 9 tables, whereas in para 2, (a) sounds like 3 tables. This makes a difference. Also, what are these tables (e.g., rater said yes/no X actually yes/no, etc)? How is "unsure" being assessed? It's not the same thing as the lack of inter-rater agreement. $\endgroup$ – gung - Reinstate Monica Jan 8 '12 at 1:39
  • $\begingroup$ @gung I am interested here in 3 2x2 tables (one per rater); they are a slice through the original larger contingency table. "Unsure" means that the rater says "I can't give you a diagnosis". $\endgroup$ – AVB Jan 8 '12 at 1:50
  • $\begingroup$ 3 tables helps. What are the 2x2 (rater can give diagnosis/can't X what?)? $\endgroup$ – gung - Reinstate Monica Jan 8 '12 at 2:10
  • $\begingroup$ @gung Columns are the two different info sources. Rows are yes/no for a rather complicated question. The tables' entries are counts (# of cases). $\endgroup$ – AVB Jan 8 '12 at 2:16
  • $\begingroup$ So, table 1 is rater A yes/no X rater B yes/no; table 2 is rater A yes/no X rater C yes/no; and table 3 is rater B yes/no X rater C yes/no; and, "no"="unsure". Do I have it right at this point? $\endgroup$ – gung - Reinstate Monica Jan 8 '12 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.