I am working on a project that compares two methods of assessing whether certain topics come up in a therapy session. I am comparing a checklist full of dichotomous (y/n) items completed by an observer who listens to the therapy session to the same checklist that the therapist completes after the therapy session. There are 5 observers who are randomly assigned therapy sessions to observe. My goal is to see the agreement between the two ways of completing the checklist.

I've thought of a few ways to do this:

Cohen's Kappa

  • Meets all assumptions except: the same two raters are not used for all observations.

Fleiss' Kappa

  • Meets all assumptions except: the targets that are being rated are not technically picked out randomly from a population. The therapists in the study choose to be in the study and were not randomly selected. But, the raters were randomly assigned to observe different sessions.

Cronbach's Alpha (Specifically Kuder-Richardson Formula 20 (KR-20))

  • I'm not sure if this technically answers the question I'm hoping to answer.

Can anyone advise on what to do? Or, if you know of any papers that do a similar thing, please send article titles. Thanks so much!

P.S. (and somewhat unrelated) Why would the Fleiss' kappa be so different than Cohen's Kappa?

  • $\begingroup$ For each session, there is only 1 observer, but it can be a different person as there are 5 observers, right? Is the therapist always the same? How many sessions are there? $\endgroup$ Commented Feb 6, 2020 at 20:40
  • $\begingroup$ Yes, there each session is only observed once by one of the five raters. There are over 40 therapist/patient pairs with around 100 sessions total. $\endgroup$
    – CJohnson
    Commented Feb 6, 2020 at 20:42
  • $\begingroup$ So there are 100 therapists? The therapist is different each time? $\endgroup$ Commented Feb 6, 2020 at 20:43
  • $\begingroup$ Sorry, I was still editing while I posted - I meant there are 100 sessions total, and 40 therapists. Some therapists appear more than once as we follow them and their patient through apx 10 sessions. We just don't have observer data on all sessions. $\endgroup$
    – CJohnson
    Commented Feb 6, 2020 at 20:44
  • $\begingroup$ And, some therapists have multiple patients, so they a certain therapist can pop up in the data more than 10 times. $\endgroup$
    – CJohnson
    Commented Feb 6, 2020 at 20:45

1 Answer 1


You may want to consider using a quasi-independence model here.

I didn't see the length of the questionnaire so I'll assume it's 30 y/n questions for this description, but that is inconsequential. If you model this as a comparison of two methods (questionnaire at end of session (M1) vs questionnaire during recorded playback (M2)) you can represent the results as counts in a 2x2x30 (M1xM2xQ) contingency table.

Assuming independence of methods, you would fit the counts with a log-linear model. However, this would likely fit poorly as we expect the raters to agree much of the time (they are listening to the same patient after all and are likely qualified to be filling out the checklist). A Quasi-Independence model adds extra terms that effectively zero out the residuals of the main diagonal of your contingency table, i.e. it fits perfectly the cells where the raters agree. This means that only the discordant results (e.g. M1 = y, M2 = n) influence the model fit. From there you can determine whether the therapist or the observer is more prone to answering "y" or "n" and if so on which question(s).

I believe the default error distribution for log-linear models in most software is the Poisson, but you can also try the Quasipoisson or Negative Binomial to add a dispersion parameter.

Note: you won't be able to assess the results of a particular therapist or observer with this structure, but if you have a broad enough sample of therapists and observers and they are randomly paired this should not be a major issue (relative to the limitations of your study).

Agresti's "An Introduction to Categorical Data Analysis" has a section on this topic (8.5.2 in the Second addition)

PSU Has a section on QI models as well.

  • $\begingroup$ Welcome to this page and thank you for your contribution. Can you give a more detailed answer? maybe refer the reader to literature? Explain your reasoning behind this suggestion? $\endgroup$
    – Jonas
    Commented Feb 7, 2020 at 9:15
  • $\begingroup$ Thanks, I've expanded the reasoning and added a few sources $\endgroup$
    – CFD
    Commented Feb 7, 2020 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.