How to use kappa() or cohen.kappa() to check for matching assigments between observers

Question

down vote favorite

Example 1. In a common situation, if we have the following data:

library("irr")
library("psych")

a<- c(1, 3, 1, 1, 2, 3, 1, 1, 3, 1, 1, 1, 1, 3, 2, 1)
d<- c(1, 3, 1, 1, 2, 3, 1, 1, 3, 1, 1, 1, 1, 3, 2, 2)
df2<- as.data.frame (cbind(a,d))
cohen.kappa(df2)

The observers "a" and "d" have assigned the observations to the same group in 15/16 cases and the cohen.kappa() would return you the following output:

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels
= levels)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries 
                 lower estimate upper 
unweighted kappa  0.68   0.89  1.00  
weighted kappa    0.96   0.96  0.96

Number of subjects = 16

Example 2. Let's say now that we have the following dataframe, that represents the same type of assignments (rows) from different observers (columns). We have three groups of subjects and we would like to know which is the overlap between the three observers assigning each subject to each of these groups.

library("irr")
library("psych")
a<- c(1, 3, 1, 1, 2, 3, 1, 1, 3, 1, 1, 1, 1, 3, 2, 1)
b<- c(2, 1, 2, 2, 3, 1, 2, 2, 1, 2, 2, 2, 2, 1, 3, 1)
c<- c(3, 1, 3, 3, 2, 1, 3, 3, 1, 3, 3, 3, 3, 1, 2, 1)

df<- as.data.frame (cbind(a,b,c))

cohen.kappa(df[,c(1,2)])
cohen.kappa(df[,c(1,3)])
cohen.kappa(df[,c(2,3)])

In our example, the subjects 1-15 have been classified within the same group by each observer, but they have different labels (i.e. each subject that was labeled with a "1" by observer "a", was labeled with a "2" by observer "b" and with a "3" by observer "c"; the same applies to other labels). The unique subject that was mislabeled was individual 16, who was labeled as a "1" by the three observers.

By this way, each pair of observers has a high degree of agreement, but it is not 100% perfect.

I would like to use either kappa2 or cohen.kappa R functions to check the observers overlap, but I always get low scores, because each observer labeled the subjects in a different way. How can I deal with this problem?

In my case, if we table the data frame we'll obtain the following output:

That is: for a=1, b=3, c=1; for a=2, b=1, c=3 and for a=3, b=2, c=2. The only mismatch is highlighted with a red circle.

In my case, the overlap between the three observers should be 93.75 (15/16) and the result of the cohen.kappa() should be the same one as for example 1, but its output for cohen.kappa(df[,c(1,2)]) for example is the following one:

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels = levels)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries 
                 lower estimate upper
unweighted kappa -0.50    -0.33 -0.16
weighted kappa   -0.44    -0.44 -0.44

Number of subjects = 16.

BTW, I cannot relabel each group, because this will be used in the context of 10000 iterations.

R version 3.3.2 (2016-10-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] magrittr_1.5       factoextra_1.0.5   cluster_2.0.6      bindrcpp_0.2       car_2.1-5         
 [6] dplyr_0.7.3        plyr_1.8.4         semTools_0.4-14    lavaan_0.5-23.1097 psych_1.7.8       
[11] pvclust_2.0-0      mclust_5.3         useful_1.2.3       ggplot2_2.2.1      irr_0.84          
[16] lpSolve_5.6.13    

loaded via a namespace (and not attached):
 [1] modeltools_0.2-21  kernlab_0.9-25     splines_3.3.2      lattice_0.20-35    colorspace_1.3-2  
 [6] viridisLite_0.2.0  stats4_3.3.2       mgcv_1.8-22        rlang_0.1.2        ggpubr_0.1.5      
[11] nloptr_1.0.4       foreign_0.8-69     glue_1.1.1         prabclus_2.2-6     fpc_2.1-10        
[16] bindr_0.1          robustbase_0.92-7  MatrixModels_0.4-1 munsell_0.4.3      gtable_0.2.0      
[21] mvtnorm_1.0-6      labeling_0.3       SparseM_1.77       quantreg_5.33      pbkrtest_0.4-7    
[26] flexmix_2.3-14     parallel_3.3.2     class_7.3-14       DEoptimR_1.0-8     trimcluster_0.1-2 
[31] Rcpp_0.12.12       diptest_0.75-7     scales_0.5.0       lme4_1.1-14        gridExtra_2.3     
[36] mnormt_1.5-5       ggrepel_0.6.5      grid_3.3.2         quadprog_1.5-5     tools_3.3.2       
[41] lazyeval_0.2.0     tibble_1.3.4       whisker_0.3-2      pbivnorm_0.6.0     pkgconfig_2.0.1   
[46] dendextend_1.5.2   MASS_7.3-47        Matrix_1.2-11      viridis_0.4.0      assertthat_0.2.0  
[51] minqa_1.2.4        R6_2.2.2           nnet_7.3-12        nlme_3.1-131      

Thank you very much,

Yatrosin

Answer 1

I have managed to solve the problem, but I guess there are better and easier ways of dealing with this issue and I would like you to tell me if you have one.

library(psych)
library(e1071)
library(car)
library(plyr)

This is the example 1. it can be analyzed with cohen.kappa() without any problem

a<- c(1, 3, 1, 1, 2, 3, 1, 1, 3, 1, 1, 1, 1, 3, 2, 1)
d<- c(1, 3, 1, 1, 2, 3, 1, 1, 3, 1, 1, 1, 1, 3, 2, 2)
D_df<- as.data.frame (cbind(a,d))
cohen.kappa(D_df)

It returns you the following output:

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels = levels)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries 
                 lower estimate upper
unweighted kappa  0.68     0.89  1.00
weighted kappa    0.96     0.96  0.96

 Number of subjects = 16 

For example 2:

a<- c(1, 3, 1, 1, 2, 3, 1, 1, 3, 1, 1, 1, 1, 3, 2, 1)
b<- c(2, 1, 2, 2, 3, 1, 2, 2, 1, 2, 2, 2, 2, 1, 3, 1)

# We generate a vector containing all possible values of the first
# observer's assignments in the order they appear
a_unique <- unique(a)
a_unique
[1] 1 3 2

# We table the first observer's assignments with the second observer's
# assignments
taba <- table(a,b)

# Using the matchClasses function of the e1071 package we check which
# values given by observer b match the ones that were assigned by
# observer a
b_match<-matchClasses(taba, "exact")

# We sort the output vector with the corresponding values between a and b
# according to the order of the observations given by observer a
b_match_sorted<- c(which(b_match==1),which(b_match==3),which(b_match==2))

# We recode the observer's b values using the original a_unique vector
# and the sorted vector for observer's b corresponding values using the
# mapvalues() from plyr
b_recoded<-mapvalues(b, from = a_unique, to = b_match_sorted)

# We generate a dataframe with the recoded b vector and the original
# a vector 
B_df<- cbind(a,b_recoded)

# Now that both vectors have the same values, cohen.kappa() can be used
# exactly in the same way as in example 1.
cohen.kappa(B_df)

The same can be applied to observer c.

c<- c(3, 1, 3, 3, 2, 1, 3, 3, 1, 3, 3, 3, 3, 1, 2, 1)
tabc <- table(a,c)
c_match<-matchClasses(tabc, "exact")
c_match_sorted<- c(which(c_match==1),which(c_match==3),which(c_match==2))
c_recoded<-mapvalues(c, from = a_unique, to = c_match_sorted)
C_df<- cbind(a,c_recoded)
cohen.kappa(B_df)

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels = levels)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries 
                 lower estimate upper
unweighted kappa  0.67     0.89  1.00
weighted kappa    0.69     0.84  0.99

Number of subjects = 16 

This result is very similar to the one observed in Example 1, as expected.

However, if anyone has an easier solution for this specific issues, I would like you to share it.

Thank you very much,

Best,

Yatrosin