A colleague and I are trying to annotate an alignment task.
We have two documents. The first document is the original copy and the other document is a modified version.
The other annotator and I are trying to align the sentences in the second copy to the original. A sentence in the second document could be the same as in the original; a sentence could also be added; and a sentence in the original could also be deleted.
Also, one sentence could be aligned to multiple sentences in the original copy. And multiple sentences could be aligned to one sentence.
Here is an example:
The first document has 10 sentences: Sent-1, Sent-2, Sent-3, Sent-4, ... , Sent-10.
The second document has 9 sentences: Sent-1, Sent-2, Sent-3, ... , Sent-9.
My annotation is: 1, 2, 3, 4, ADD, 5, 6, 8, 10
It shows that I think Sent-1, 2, 3,4 are aligned to 1,2,3,4 in the original sentence, Sent-5 is added, Sent-6 is aligned to 5, and Sent-7 to 6, Sent-8 to 8, Sent-9 to 10. The other unaligned ones are deleted.
The other annotator has: 1, 2, 3, 4, 5, 5, 6, 9, 10
This shows that the other annotator has a different opinion on sentence 5, and sentence 8. He thinks 5 should be aligned to 5 and 8 should be aligned to 9.
So what is the kappa here?