# How to test the inter-rater reliability of a nominal coding process that generates ordinal data?

A researcher studying hunger on a tropical island asks respondents, "What did you eat for lunch yesterday?" and the possible choices are "(4) Nothing; (3) a banana; (2) an unripe coconut; (1) a ripe coconut."

The researcher then says, my categories are 4, 3, 2, 1 and should be understood as ordinal with 4 being "most hungry." To simplify, we grant this ordinal interpretation.

However, now we need to validate this study. Is the coding process accurate and reliable? My problem is, should I measure the reliability of the coding process by treating the data as nominal or ordinal?

It seems to me that the coding process is nominal while the outputs receive an ordinal interpretation. Each respondent takes the phenomenon or coding unit "what I ate for lunch yesterday" and asks "does it match the nominal category "nothing,".. or "ripe coconut?" None of these categories possesses an intrinsic numerical or ranked relationship with any other (unlike the questions, "did you eat a small, medium, or large coconut for lunch," or "how many coconuts did you eat yesterday?"). Nonetheless, the researcher wants to interpret the data as ordinal and this is granted.

To focus on specifics, Krippendorff's alpha allows this researcher to calculate an inter-rater reliability statistic on nominal, ordinal, interval, ratio, and other types of data. Assume that Krippendorff's alpha is 0.6 (unreliable) when data is treated nominally and 0.9 (very reliable) when data is treated ordinally. Which one should the researcher reject and why? I think 0.9 must be rejected and 0.6 is correct.

• Krippendorf's alpha is a measure of agreement, not reliability. (f you have two raters, that doesn't matter, if you have more, it does.) – Jeremy Miles Oct 30 '19 at 2:07