Cohen's kappa and confounders

Question

I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee.

For example, age, gender, traffic intensity, and number of previous failed tests (all known to both examiners) may affect the examiner decision and possibly the agreement with the other examiner.

One can simply do several sub-group analyses, but I'm not sure this is a good method.

How would you evaluate the effects of such factors on a measure of concordance (e.g., Cohen's kappa)?

Is it possible to "adjust" Cohen's kappa for such factors?

Answer 1

I don't think kappa is the route to go here, because that requires calculation over multiple "ratees" (here drivers). For each particular driver, the agreement can be either a) One of two things or b) One of three things or c) One of four things, depending on what your interest is:

A. One of two things - agree or disagree. Then you could use this as the dependent variable and use logistic regression.

B. One of three things - both pass, both fail, disagree. Then you would use multinomial logistic or maybe ordinal logistic.

C. One of four things - both pass, both fail, A pass B fail, A fail B pass. Here you would use multinomial logistics.

For each driver you would have the DV (one of the three just described) and whatever IVs you wanted.

Of course, if you have multiple pairs of examiners, things get more complicated.