2
$\begingroup$

Krippendorff has a paper (free and official versions) describing an adaptation of his measure of reliability α to continuous data, such as text. Specifically, it calculates the reliability of both unitizing (i.e., how the text is split into coded and uncoded units) and coding (i.e., which codes are assigned to what units).

I'm wondering if this has been implemented, say, in R. In his paper, Krippendorff mentions that a "computer program for calculating these [adaptations of α] is currently being developed." However, I could find no mention of or link to such software, neither on Krippendorff's site nor elsewhere. The paper is detailed enough that it would allow me to implement it myself, but if it's already been done no sense in reinventing the wheel.

Thanks in advance.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ How is text "continuous data"? Please explain. $\endgroup$
    – Nick Cox
    Sep 2, 2013 at 14:21
  • $\begingroup$ Krippendorff treats text as continuous in that is not necessarily divided into pre-defined units. Yes, there are words, sentences, paragraphs, etc., but a coder might choose to unitize along any of these. Other examples of "continuous" data include video, audio, or "anything that has an extension in a measurable dimension" (Krippendorff 2004, p. 790). $\endgroup$
    – EricPSB
    Sep 3, 2013 at 15:24
  • 1
    $\begingroup$ Thanks for the clarification. I guess most statistical people would think of "continuous" as referring to measurements on a continuous numerical scale, e.g. height or weight or temperature. That's clearly a different sense. $\endgroup$
    – Nick Cox
    Sep 3, 2013 at 15:27

3 Answers 3

Reset to default
2
$\begingroup$

There is also now this new paper which extends Krippendorff's work on unitizing:

https://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00227 ("The Unified and Holistic Method Gamma (γ) for Inter-Annotator Agreement Measure and Alignment" by Yann Mathet, Antoine Widl̈ocher, Jean-Philippe Ḿetivier in Computational Linguistics Journal (2015))

The code is in Java: https://gamma.greyc.fr/

Followup: https://www.mitpressjournals.org/doi/full/10.1162/COLI_a_00296 ("The Agreement Measure γcat a Complement to γ Focused on Categorization of a Continuum" by Yann Mathet in Computational Linguistics Journal (2017))

There are certain weakness(es) of alpha that were addressed in these two papers. I'm still reading them, so I don't know what these weaknesses were and how they were addressed. I'll hopefully be able to say more when I'm done reading.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Welcome to the site. We are trying to build a permanent repository of high-quality statistical information in the form of questions & answers. Thus, we're wary of link-only answers, due to linkrot. Can you post a full citation & a summary of the information at the link, in case it goes dead? $\endgroup$
    – Jan Kukacka
    May 15, 2018 at 6:25
  • $\begingroup$ Yes, these γ functions are exactly what I had in mind, as indicated by the following passage from the 2017 paper abstract. "Agreement on unitizing, where several annotators freely put units of various sizes and categories on a continuum, is difficult to assess because of the simultaneaous discrepancies in positioning and categorizing. The recent agreement measure γ offers an overall solution that simultaneously takes into account positions and categories." $\endgroup$
    – EricPSB
    Aug 2, 2019 at 18:44
4
$\begingroup$

Krippendorff's alpha has been implemented in R via the irr package and kripp.alpha function. It can be used for continuous data. You can find an example here. Just use a continuous rating and change the method argument to interval or ratio.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I don't think that's what the OP means by continuous data. $\endgroup$
    – Jeremy Miles
    Jul 27, 2019 at 20:13
1
$\begingroup$

For readers who are searching papers for this "implementation" problem as it is named by EricPSB:

Answering the Call for a Standard Reliability Measure for Coding Data. Andrew F. Hayes and Klaus Krippendorff. Accessed via http://afhayes.com/public/cmm2007.pdf SPSS MACRO available at: http://www.afhayes.com/public/kalpha.zip

An easy read documentation on the use of this MACRO: Calculating inter-coder reliability in media content analysis using Krippendorff’s Alpha. Knut De Swert (2012). Accessed via https://www.polcomm.org/wp-content/uploads/ICR01022012.pdf

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.