I am occasionally presented with data where a rating scale was used (i.e. options 1 through 5). Sometimes these are ordinal, e.g.

  1. Very positive.
  2. Somewhat positive
  3. Neutral
  4. Somewhat negative
  5. Very negative

But sometimes these are not:

  1. Yes.
  2. No.
  3. Wrong domain.
  4. Wrong language.
  5. Not enough information.

The information about the rating scale is not always available, or not always provided (because the user forgot to say), nonetheless, I'd like to calculate agreement amongst respondents, using Cohen's Kappa (or, equivalently, Krippendorff's Alpha).

An approach that I'm considering is to calculate agreement, using weighted (assuming the data are ordinal) and unweighted (assuming the data are nominal), and then take the higher, assuming that this is correct. (It may also have implications for later analysis - if the ordinal value is higher, we assume that the data are ordinal.

This seems a little risky, but perhaps better than nothing?

If anyone had ideas that could help me to clarify my thinking, I'd appreciate it.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy