# How to calculate this kind of kappa?

A colleague and I are trying to annotate an alignment task.

We have two documents. The first document is the original copy and the other document is a modified version.

The other annotator and I are trying to align the sentences in the second copy to the original. A sentence in the second document could be the same as in the original; a sentence could also be added; and a sentence in the original could also be deleted.

Also, one sentence could be aligned to multiple sentences in the original copy. And multiple sentences could be aligned to one sentence.

=======

Here is an example:

The first document has 10 sentences: Sent-1, Sent-2, Sent-3, Sent-4, ... , Sent-10.

The second document has 9 sentences: Sent-1, Sent-2, Sent-3, ... , Sent-9.

My annotation is: 1, 2, 3, 4, ADD, 5, 6, 8, 10

It shows that I think Sent-1, 2, 3,4 are aligned to 1,2,3,4 in the original sentence, Sent-5 is added, Sent-6 is aligned to 5, and Sent-7 to 6, Sent-8 to 8, Sent-9 to 10. The other unaligned ones are deleted.

The other annotator has: 1, 2, 3, 4, 5, 5, 6, 9, 10

This shows that the other annotator has a different opinion on sentence 5, and sentence 8. He thinks 5 should be aligned to 5 and 8 should be aligned to 9.

So what is the kappa here?