I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee.
For example, age, gender, traffic intensity, and number of previous failed tests (all known to both examiners) may affect the examiner decision and possibly the agreement with the other examiner.
One can simply do several sub-group analyses, but I'm not sure this is a good method.
How would you evaluate the effects of such factors on a measure of concordance (e.g., Cohen's kappa)?
Is it possible to "adjust" Cohen's kappa for such factors?