Estimate sensitivity and specificity from inter-rater agreement?

Question

Assuming two binary (Y in {0, 1}) annotators or classifiers (A and B), that are:

1. Conditionally independent, i.e. P(A=0, B=0|Y=1) = P(A=0|Y=1)*P(B=0|Y=1) and the same for Y=0.
2. Better than random, i.e. ROC-AUC>0.5

Does high Cohen's kappa imply high sensitivity and specificity?

Is it possible to estimate (or at least place a lower bound) on the sensitivity and specificity of each classifier (or their ensemble) from the Cohen Kappa?

My intuition is that if both assumptions hold, then to agree the models must both conform to the ground truth.

I could not find any literature about direct estimation of accuracy metrics from agreement metrics.

Answer 1

Does high Cohen's kappa imply high sensitivity and specificity?

Answer: No.

Is it possible to estimate (or at least place a lower bound) on the sensitivity and specificity of each classifier (or their ensemble) from the Cohen Kappa?

Answer: No.

Kappa is more like classification accuracy and very different from sensitivity and specificity. Kappa is based on counts that load on the diagonal of the interrater agreement matrix. This is very similar to the confusion matrix in class prediction, where the rows and columns are observed and predicted class labels, respectively.

Sensitivity and specificity are not normally associated with Kappa, rather, they are typically determined via methods used in "diagnostic screening" for multiway tables. The table below for >2 classifiers shows that there is a sens and spec value for each class label in the 3-class Fisher Iris dataset.

There's also a terminology issue with your question. The target class labels are binary (0,1) or (1,2); classifiers aren't typically called binary, multinomial, etc.