I have a question about the inter-annotator reliability measure of Krippendorff's alpha for text continua or text unitization. The measure is explained in detail in the paper.

I have texts and 2 coders/annotators. The task of the annotators is to identify noun phrases in those texts using the maximal principle; that is to say, all the elements that modify the head noun need to be tagged in the same span of the noun phrase. Of course, there is disagreement between the two coders and I need to measure reliability.

I read Krippendorff's paper and my question is: how to compute alpha in the case in the picture: one unit by the first annotator maps to multiple separate units with gaps in the second annotator's output? This case propagates lots of disagreement in subsequent units.

In the paper, all the examples given correspond one-to-one but with different beginnings and lengths, but nothing like my example :/

Are my computations (in the picture) correct?

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.