3
$\begingroup$

A researcher studying hunger on a tropical island asks respondents, "What did you eat for lunch yesterday?" and the possible choices are "(4) Nothing; (3) a banana; (2) an unripe coconut; (1) a ripe coconut."

The researcher then says, my categories are 4, 3, 2, 1 and should be understood as ordinal with 4 being "most hungry." To simplify, we grant this ordinal interpretation.

However, now we need to validate this study. Is the coding process accurate and reliable? My problem is, should I measure the reliability of the coding process by treating the data as nominal or ordinal?

It seems to me that the coding process is nominal while the outputs receive an ordinal interpretation. Each respondent takes the phenomenon or coding unit "what I ate for lunch yesterday" and asks "does it match the nominal category "nothing,".. or "ripe coconut?" None of these categories possesses an intrinsic numerical or ranked relationship with any other (unlike the questions, "did you eat a small, medium, or large coconut for lunch," or "how many coconuts did you eat yesterday?"). Nonetheless, the researcher wants to interpret the data as ordinal and this is granted.

To focus on specifics, Krippendorff's alpha allows this researcher to calculate an inter-rater reliability statistic on nominal, ordinal, interval, ratio, and other types of data. Assume that Krippendorff's alpha is 0.6 (unreliable) when data is treated nominally and 0.9 (very reliable) when data is treated ordinally. Which one should the researcher reject and why? I think 0.9 must be rejected and 0.6 is correct.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Krippendorf's alpha is a measure of agreement, not reliability. (f you have two raters, that doesn't matter, if you have more, it does.) $\endgroup$
    – Jeremy Miles
    Commented Oct 30, 2019 at 2:07

1 Answer 1

Reset to default
1
$\begingroup$

To make the argument that these categories have an ordered relationship, evidence of such a relationship (e.g., from caloric value or agreed-upon social meaning) seems necessary. In the absence of such evidence, a nominal interpretation is most appropriate. Krippendorff's alpha can indeed handle different weighting schemes, but to be clear so can all other chance-adjusted indices. Obviously a score of 0.9 is better than 0.6 but it may be that a score of 0.6 is still "good enough." Reliability is not strictly dichotomous.

Edit: I'm also confused as to why the authors didn't just directly ask about hunger if that is their main interest...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.