I have 20 sequences that were coded by 2 judges, for example: Sequence 1, Judge 1: A-B-C-A-B-E-F Sequence 1, Judge 2: A-B-E-C-A-B-F I want to check the inter-rater reliability. Any suggestions on how to do this?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ You could measure the (dis)similarity between the two sequences. However, we would need more information about the sequences to chose the dissimilarity measure: What are the states? Is the alignment between the positions in the two sequences important or is time warp allowed? etc. $\endgroup$– GilbertCommented Apr 17, 2017 at 11:05
-
$\begingroup$ The states are web moves (i.e. click, search, browse, etc.) performed by a user. The alignment is important as the two sequences should be exactly the same states and in the same sequence. $\endgroup$– user3315563Commented Apr 18, 2017 at 14:23
-
$\begingroup$ Are TramineR's dissimilarity indices comparable with Cohen's Kappa, or any other standard IRR's? $\endgroup$– user3315563Commented Apr 21, 2017 at 14:22
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
Since you want to make position wise comparison, you should use the Hamming metric. I illustrate with five example sequences. The first two are the two examples you are giving. The 3rd one has no match with any of the first two. The 4th is the same as the 1st one, and the 5th differs from the first one only at one position.
library(TraMineR)
s.data <- c("A-B-C-A-B-E-F",
"A-B-E-C-A-B-F",
"B-A-B-E-F-A-B",
"A-B-C-A-B-E-F",
"A-B-C-A-B-E-B")
s <- seqdef(s.data)
diss <- seqdist(s, method="HAM", norm=TRUE)
round(diss, digit=2)
## [,1] [,2] [,3] [,4] [,5]
## [1,] 0.00 0.57 1.00 0.00 0.14
## [2,] 0.57 0.00 1.00 0.57 0.71
## [3,] 1.00 1.00 0.00 1.00 0.86
## [4,] 0.00 0.57 1.00 0.00 0.14
## [5,] 0.14 0.71 0.86 0.14 0.00
We see that the 5th sequence is closer from the first sequence than the 2nd one. Also we see that the 3th sequence completely disagrees with 1 and 2, and that the 4th completely agrees with the first.
