A colleague and I are trying to annotate an alignment task.

We have two documents. The first document is the original copy and the other document is a modified version.

The other annotator and I are trying to align the sentences in the second copy to the original. A sentence in the second document could be the same as in the original; a sentence could also be added; and a sentence in the original could also be deleted.

Also, one sentence could be aligned to multiple sentences in the original copy. And multiple sentences could be aligned to one sentence.


Here is an example:

The first document has 10 sentences: Sent-1, Sent-2, Sent-3, Sent-4, ... , Sent-10.

The second document has 9 sentences: Sent-1, Sent-2, Sent-3, ... , Sent-9.

My annotation is: 1, 2, 3, 4, ADD, 5, 6, 8, 10

It shows that I think Sent-1, 2, 3,4 are aligned to 1,2,3,4 in the original sentence, Sent-5 is added, Sent-6 is aligned to 5, and Sent-7 to 6, Sent-8 to 8, Sent-9 to 10. The other unaligned ones are deleted.

The other annotator has: 1, 2, 3, 4, 5, 5, 6, 9, 10

This shows that the other annotator has a different opinion on sentence 5, and sentence 8. He thinks 5 should be aligned to 5 and 8 should be aligned to 9.

So what is the kappa here?


You tell us! (as it were).

Seriously, it's your choice, but even as a amateur in this field I note

  1. Scoring from the point of the view of the second document rather than the first is a choice. If you score from the point of the view of the original document you might declare sentences DELETEd as well as ADDed and you have 10 sentences, so for those reasons alone you are likely to get different results for kappa. This may be a standard situation in some fields, but in applications of kappa I've seen the comparison is symmetric (e.g. radiologists are comparing the same set of X-rays).

  2. There are it seems two reasons for disagreement, that you identify different sentences as sources and that one of you identifies a sentence as ADD but the other doesn't. It's not at all clear that these reasons are equally serious; conversely, it's a research decision how to score them.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.