1
$\begingroup$

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance.

However, I've noticed that when subjects are fairly homogeneous, $\kappa$ drops precipitously and seems to lose any relation to a lay understanding of agreement.

To give this some real world flavor, say I have ratings for depression, anxiety, and no-diagnosis, and I want to see if raters are especially good or bad at detecting depression in just the patients I know have depression. I pull out a subset of ratings based on this criterion, and use Fleiss's $\kappa$:

library(irr)

low_var<-data.frame(
    r1=round(rnorm(250,1,.2)),
    r2=round(rnorm(250,1,.2)),
    r3=round(rnorm(250,1,.2)),
    r4=round(rnorm(250,1,.2)),
    r5=round(rnorm(250,1,.2)))

head(low_var)

##   r1 r2 r3 r4 r5
## 1  1  1  1  1  1
## 2  1  1  1  2  1
## 3  0  1  1  1  1
## 4  1  1  1  0  1
## 5  1  2  1  1  1
## 6  1  1  1  1  1

kappam.fleiss(low_var)

##  Fleiss' Kappa for m Raters
## 
##  Subjects = 250 
##    Raters = 5 
##     Kappa = -0.0103 
## 
##         z = -0.653 
##   p-value = 0.514 

I understand in a vague sort of way that if you don't have people who are not depressed in your sample, you can't say anything about how good raters are at distinguishing between things. But they certainly agree. Am I just mixing up ideas of reliability and validity here? Set me on the right path, folks. Many thanks for your input.

The questions

  1. Can you help me understand better why, mathematically, the Fleiss $\kappa$ decreases with less variability in categorization.
  2. Why is this a desirable property of the statistic?
$\endgroup$
2
$\begingroup$

To help you understand, I am going to simplify things by moving to dichotomous coding from two raters. However, rest assured that the same basic idea will also apply to the generalized formulas.

Kappa is the ratio of observed non-chance agreement to possible non-chance agreement:

$$\kappa = \frac{p_o-p_c}{1-p_c}$$

where $p_o$ is the percent observed agreement and $p_c$ is the percent chance agreement.

If we set up a two-by-two contingency table for raters $A$ and $B$, we get:

$$ \begin{array}{|c|c|c|} \hline ~&A=1&A=0\\ \hline B=1&a&b\\ \hline B=0&c&d\\ \hline \end{array} $$

In this table, the cells $a,b,c,d$ are counts for how often each rater assigned an item to each category. For instance, $a$ is the number of times that both raters $A$ and $B$ assigned an item to category $1$. From here, we can calculate the percent observed agreement:

$$p_o=\frac{a+d}{a+b+c+d}$$

So percent observed agreement is just the proportion of items that both raters assigned to the same category. Now that we understand this, we can also see that $p_o$ is not affected by the heterogeneity of the data. If all of the items were in cell $a$, then $p_o$ would be the same as if half the items were in cell $a$ and the other half were in cell $d$. But if $\kappa$ isn't being influenced by heterogeneity because of $p_o$, then why is it? If we return to the formula for $\kappa$, we can see that the only other option is $p_c$.

$$ \begin{array}{|c|c|c|c|} \hline ~&A=1&A=0&\text{Total}\\ \hline B=1&a&b&g_1\\ \hline B=0&c&d&g_2\\ \hline \text{Total}&f_1&f_2&n\\ \hline \end{array} $$

Above I have included an expanded version of the contingency table that now includes marginal totals (e.g., $g_1=a+b$) and a grand total (i.e., $n=a+b+c+d$). These totals will simplify the notation we will be using to estimate percent chance agreement. Before that, however, I want to make it explicit that there are multiple ways to calculate percent chance agreement and thus multiple versions of the kappa coefficient. I will begin with Cohen's (1960) kappa coefficient because it is simplest. I will then move on to Fleiss' (1971) kappa coefficient because you asked about it specifically.

Cohen's kappa estimates percent chance agreement by using Baye's rule. It assumes that the marginal probabilities are consistent within each rater. From this assumption, it follows that the probability of each cell occurring by chance is equal to the product of its marginal probabilities. The expected percent chance agreement would thus be the probability of agreeing by chance on category $1$ OR agreeing by chance on category $0$, i.e., the sum of the probabilities of cells $a$ and $d$.

$$\text{Cohen's }p_c=\Bigg(\frac{f_1}{n}\Bigg)\Bigg(\frac{g_1}{n}\Bigg) + \Bigg(\frac{f_2}{n}\Bigg)\Bigg(\frac{g_2}{n}\Bigg)$$

Fleiss' kappa estimates percent chance agreement slightly differently. Borrowing the assumptions of Scott (1955), it assumes that the marginal probabilities are consistent across raters on average. From this assumption, it follows that the probability of each cell occurring by chance is equal to the square of its average marginal probability. The expected percent chance agreement is thus:

$$\text{Scott's }p_c=\Bigg(\frac{(f_1+g_1)/2}{n}\Bigg)^2 + \Bigg(\frac{(f_2+g_2)/2}{n}\Bigg)^2$$

Now we have the information needed to finally answer your question. If one of the categories is much more likely than the other (i.e., if the data is highly homogeneous), then some of the marginal probabilities are going to be very high and $p_c$ is going to also become very high. This, in turn, will make $\kappa$ much lower by virtue of the subtraction in the numerator of the $\kappa$ formula above.

In short, a lack of heterogeneity produces a high estimate of chance agreement given the kappa coefficient's assumptions. This, in turn, produces a lower kappa score. This fact becomes more apparent when you apply a generalized kappa formula to the example data in your question:

Number of items = 6
Number of raters = 5
Possible categories = [0;1;2]
Observed categories = [0;1;2]
Scale of measurement = nominal
Percent observed agreement = 0.733
Percent chance agreement = 0.760

Fleiss' kappa coefficient = -0.111

Whether or not this behavior is desirable depends on whether or not the assumptions made by these measures match your data. Are your raters truly consistent in their marginal probabilities, either individually or on average? Do they truly engage in a chance-based process that is independent of the data itself? If not, then this behavior is erroneous and the kappa score will be problematically low or high. Zhao, Liu, & Deng (2012) offer a very insightful review and discussion of these issues.

If you would like to learn more about these and other measures of inter-rater reliability and find functions for their calculation, please visit and explore my mReliability website.

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.

Scott, W. A. (1955). Reliability of content analysis: The case of nominal scaling. Public Opinion Quarterly, 19(3), 321–325.

Zhao, X., Liu, J. S., & Deng, K. (2012). Assumptions behind inter-coder reliability indices. In C. T. Salmon (Ed.), Communication Yearbook (pp. 418–480). Routledge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.