free-marginal multirater/multicategories agreement indexes and the K categories PABAK

Question

A wide variety of ways to access the level of agreement between two or more raters was proposed. One of the most common ways is through metrics that describe characteristics of a Confusion matrix (contigence table), such as the Kappa index.

According to the specific requirements of each application were developed several "Kappa like indices" which allow to evaluate two or more judges (biraters or multiraters, multicategory). Those indices can be fixed or free-marginal types and may or may not be influenced by bias or prevalence:

Randolph (2005) - Free-Marginal Multirater Kappa (multirater κfree): An Alternative to Fleiss’ FixedMarginal Multirater Kappa

In this context, I'm looking for the reference of a specific index (the multicategories PABAK) and I am also interested in other "free-marginal multirater/multicategories indexes". It's for an aplication on mapping quality assessment.

The original PABAK index can only be applied to a 2x2 matrix and was proposed by Byrt and Bishop (1993). It is defined as:

Matrix

a b

c d

PABAK=([x/(x+y)]-0.5)/(1-0.5)=2*Po-1

where:

• Po = (a+d)/n

• n = a + b + c + d

• x=b+c/2

• y=a+d/2

The adapted PABAK Index for K categories is given by:

Kp= ((K*Po)-1)/(K-1)

This index was mentioned on this link:

Adjusting kappa inter-rater agreement for prevalence

and it was also used in this online calculator:

http://www.singlecaseresearch.org/calculators/pabak-os

However, I was not able to find the source of this adaptation or a demonstration up to now.

Besides that, I would like to know if is there any related adaptation of the Byrt and Bishop's prevalence and bias for K categories?

If you know the source of the K categories PABAK or know other free-marginal multirater/multicategories agreement indexes, please leave the answer and its reference.

Thank you.

Answer 1

The logic behind PABAK and equivalent formulations of the metric have been proposed in many different forms over the years in many different fields. The first was actually well before Byrt and Bishop (1993): it was Bennett, Alpert, and Goldstein (1954). I have no idea why Randolph's table claims that the S score assumed fixed marginals; it is clearly equivalent to PABAK. The most complete generalization of the formula allows it to accommodate any number of raters, any number of categories, missing data, and non-nominal weighting of categories. You can find this in Gwet (2014) where it is called kappa-q and kappa-BP alternatively. I prefer to call it the generalized S score, as Bennett, Alpert, and Goldstein originally named their version the S score.

A simpler version that extends S for multiple categories (but can't handle non-nominal weights) is:

$$S=\frac{p_o-1/q}{1-1/q}$$

where $q$ is the number of categories and $p_o$ is the observed percent agreement.

References

Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communication through limited response questioning. The Public Opinion Quarterly, 18(3), 303–308.

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.