I am trying to calculate inter-rater reliability for my data. I have multiple participants and multiple responses per participant. Each participant provided a list of responses, which will then be rated on a pass/fail basis by two raters. An example of what the data would look like

I have only seen examples Cohen's kappa being used in situations where there is one response that is being rated per participant. Is it acceptable to use Cohen's kappa in my situation or does is violate some assumptions? If so, what would be a better alternative?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.