0
$\begingroup$

I have a question about using alpha to measure Krippendorff's inter-annotator reliability in qualitative data. After reading the papers, I'm still not sure how to handle this case.

I'm working on a linguistics project. The annotators are given a text (in Arabic actually) and they need to mark noun phrases following the maximal principle; that is to say, all the elements that modify the head noun need to be included in the span of the noun phrase. Of course, annotators sometimes disagree as to whether a specific element should or shouldn't be included.

The case I am asking about is this:

Give a sentence like "I was the youngest girl with blue eyes in the workshop"

Annotator 1 tags it as such where noun phrases are marked by brackets.

[I] was [the youngest girl with blue eyes in the workshop].

Annotator 2 tags it as [I] was [the youngest girl] with [blue eyes] in [the workshop]

So, one unit by annotator one has been split into three units by annotator two. To visualize it with units of length it will be like:

enter image description here

My question is how to calculate the squared difference function and therefore the observed disagreement? This is what I did. Are my calculations correct?

the youngest girl with blue eyes in the workshop vs. the youngest girl

b1 3 l1 9

b2 3 l2 3

δ2 = (3 - 3)2 + (3 + 9 – 3 - 3)2 = 36

the youngest girl with blue eyes in the workshop vs. blue eyes

b1 3 l1 9

b2 7 l2 2

δ2 = (3 – 7)2 + (3 + 9 – 7 – 2)2 = 16 + 9 = 25?

the youngest girl with blue eyes in the workshop vs. the workshop

b1 3 l1 9

b2 10 l2 2

δ2 = (3 – 10)2 + (3 + 9 – 10 – 2)2 = 49?

m = 2

L = 11

Doc = 36 + 25 + 49 / 2(2-1)112 = 110 / 242 = 0.455

$\endgroup$
  • $\begingroup$ Another question I have is what if the unit has only one word: should I count the length starting from 0 or 1? $\endgroup$ – Sabba May 18 '19 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.