I am working on a project that compares two methods of assessing whether certain topics come up in a therapy session. I am comparing a checklist full of dichotomous (y/n) items completed by an observer who listens to the therapy session to the same checklist that the therapist completes after the therapy session. There are 5 observers who are randomly assigned therapy sessions to observe. My goal is to see the agreement between the two ways of completing the checklist.
I've thought of a few ways to do this:
Cohen's Kappa
- Meets all assumptions except: the same two raters are not used for all observations.
Fleiss' Kappa
- Meets all assumptions except: the targets that are being rated are not technically picked out randomly from a population. The therapists in the study choose to be in the study and were not randomly selected. But, the raters were randomly assigned to observe different sessions.
Cronbach's Alpha (Specifically Kuder-Richardson Formula 20 (KR-20))
- I'm not sure if this technically answers the question I'm hoping to answer.
Can anyone advise on what to do? Or, if you know of any papers that do a similar thing, please send article titles. Thanks so much!
P.S. (and somewhat unrelated) Why would the Fleiss' kappa be so different than Cohen's Kappa?