Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The Kappa ($\kappa$) statistic was introduced in 1960 by Cohen [1] to measure agreement between two raters. Its variance, however, had been a source of contradictions for quite a some time.

My question is about which is the best variance calculation to be used with large samples. I am a inclined to believe the one tested and verified by Fleiss [2] would be the right choice, but this does not seems to be the only published one which seems to be correct (and used thoroughly fairly recent literature).

Right now I have two concrete ways to compute its asymptotic large sample variance:

  • The corrected method published by Fleiss, Cohen and Everitt [2];
  • The delta method which can be found in the book by Colgaton, 2009 [4] (page 106).

To illustrate some of this confusion, here is a quote by Fleiss, Cohen and Everitt [2], emphasis mine:

Many human endeavors have been cursed with repeated failures before final success is achieved. The scaling of Mount Everest is one example. The discovery of the Northwest Passage is a second. The derivation of a correct standard error for kappa is a third.

So, here is a small summary of what happened:

  • 1960: Cohen publishes his paper "A coefficient of agreement for nominal scales" [1] introducing his chance-corrected measure of agreement between two raters called $\kappa$. However, he publishes incorrect formulas for the variance calculations.
  • 1968: Everitt attempts to correct them, but his formulas were incorrect as well.
  • 1969: Fleiss, Cohen and Everitt publish the correct formulas in the paper "Large Sample Standard Errors Of Kappa and Weighted Kappa" [2].
  • 1971: Fleiss publishes another $\kappa$ statistic (but a different one) under the same name, with incorrect formulas for the variances.
  • 1979: Fleiss Nee and Landis publish the corrected formulas for Fleiss' $\kappa$.

At first, consider the following notation. This notation implies the summation operator should be applied to all elements in the dimension over which the dot is placed:

$\ \ \ p_{i.} = \displaystyle\sum_{j=1}^{k} p_{ij}$ $\ \ \ p_{.j} = \displaystyle\sum_{i=1}^{k} p_{ij}$

Now, one can compute Kappa as:

$\ \ \ \hat\kappa = \displaystyle\frac{p_o-p_c}{1-p_e}$

In which

$\ \ \ p_o = \displaystyle\sum_{i=1}^{k} p_{ii}$ is the observed agreement, and

$\ \ \ p_c = \displaystyle\sum_{i=1}^{k} p_{i.} p_{.i}$ is the chance agreement.

So far, the correct variance calculation for Cohen's $\kappa$ is given by:

$\ \ \ \newcommand{\var}{\mathrm{var}}\widehat{\var}(\hat{\kappa}) = \frac{1}{N(1-p_c)^4} \{ \displaystyle\sum_{i=1}^{k} p_{ii}[(1-p_o) - (p_{.i} + p_{i.})(1-p_o)]^2 \\ \ \ \ + (1-p_o)^2 \displaystyle\sum_{i=1}^{k} \displaystyle\sum_{j=1 \atop i\not=j}^{k} p_{ij} (p_{.i} + p_{j.})^2 - (p_op_c-2p_c+p_o)^2 \} $

and under the null hypothesis it is given by:

$\ \ \ \widehat{\var}(\hat{\kappa}) = \frac{1}{N(1-p_c)^2} \{ \displaystyle\sum_{i=1}^{k} p_{.i}p_{i.} [1- (p_{.i} + p_{i.})^2] + \displaystyle\sum_{i=1}^{k} \displaystyle\sum_{j=1, i\not=j}^{k} p_{.i}p_{j.}(p_{.i} + p_{j.})^2 - p_c^2 \} $

Congalton's method seems to be based on the delta method for obtaining variances (Agresti, 1990; Agresti, 2002); however I am not sure on what the delta method is or why it has to be used. The $\kappa$ variance, under this method, is given by:

$\ \ \ \widehat{\var}(\hat{\kappa}) = \frac{1}{n} \{ \frac{\theta_1 (1-\theta_1)}{(1-\theta_2)^2} + \frac{2(1-\theta_1)(2\theta_1\theta_2-\theta_3)}{(1-\theta_2)^3} + \frac{(1-\theta_1)^2(\theta_4-4\theta_2^2)}{(1-\theta_2)^4} \} $

in which

$\ \ \ \theta_1 = \frac{1}{n} \displaystyle\sum_{i=1}^{k} n_{ii}$

$\ \ \ \theta_2 = \frac{1}{n^2} \displaystyle\sum_{i=1}^{k} n_{i+}n_{+i}$

$\ \ \ \theta_3 = \frac{1}{n^2} \displaystyle\sum_{i=1}^{k} n_{ii}(n_{i+} + n_{+i})$

$\ \ \ \theta_4 = \frac{1}{n^3} \displaystyle\sum_{i=1}^{k} \displaystyle\sum_{j=1}^{k} n_{ij}(n_{j+} + n_{+i})^2$

(Congalton uses a $+$ subscript rather than a $.$, but it seems to mean the same thing. In addition, I am supposing that $n_{ij}$ should be a counting matrix, i.e. the confusion matrix before being divided by the number of samples as related by the formula $p_{ij} = \frac{n_{ij}}{\mathrm{samples}}$)

Another weird part is that Colgaton's book seems to refer the original paper by Cohen, but does not seems to cite the corrections to the Kappa variance published by Fleiss et al, not until he goes on to discuss weighted Kappa. Perhaps his first publication was written when the true formula for kappa was still lost in confusion?

Is somebody able to explain why those differences? Or why would someone use the delta method variance instead of the corrected version by Fleiss?

[1]: Fleiss, Joseph L.; Cohen, Jacob; Everitt, B. S.; Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, Vol 72(5), Nov 1969, 323-327. doi: 10.1037/h0028106

[2]: Cohen, Jacob (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1): 37–46. DOI:10.1177/001316446002000104.

[3]: Alan Agresti, Categorical Data Analysis, 2nd edition. John Wiley and Sons, 2002.

[4]: Russell G. Congalton and Green, K.; Assessing the Accuracy of Remotely Sensed Data: Principles and Practices, 2nd edition. 2009.

share|improve this question
some of your parentheses are off, can you please fix them? Also, you might want to format the nested parentheses as { [ (x+y)^z + a]^ b - c } to make them more readable. – StasK Jun 25 '12 at 17:05
also, please give $\kappa$ itself, and alternative equivalent formulations if they exist. Depending on the specific alternative formulations, the variance expressions may be easier to get. (I am thinking of the Gini index, for which there are five or so formulations for i.i.d. data that imply totally different variance estimators for complex survey data.) – StasK Jun 25 '12 at 17:07
Thanks for the feedback. I have corrected the formulas and added how Kappa is computed. The Kappa formulation seems consistent across the literature, only its variance doesn't. – Cesar Jun 25 '12 at 18:32
By the way, I just noticed what seems to be a printing error on Colgaton's book: he defines $p_c = \sum_{i=1}^k p_{i+} p_{+j}$, but this $j$ comes from nowhere. I suppose it was meant to be $p_c = \sum_{i=1}^k p_{i+} p_{+i}$, otherwise I am not sure it makes much sense. – Cesar Jun 25 '12 at 18:35
I can at least give you a hand with this part: "I am not sure on what the delta method is" -- and the variance there comes from here – Glen_b Dec 25 '12 at 23:00

I don't know which of the two ways to calculate the variance is to prefer but I can give you a third, practical and useful way to calculate confidence/credible intervals by using Bayesian estimation of Cohen's Kappa.

The R and JAGS code below generates MCMC samples from the posterior distribution of the credible values of Kappa given the data.


# Creating some mock data
rater1 <- c(1, 2, 3, 1, 1, 2, 1, 1, 3, 1, 2, 3, 3, 2, 3) 
rater2 <- c(1, 2, 2, 1, 2, 2, 3, 1, 3, 1, 2, 3, 2, 1, 1) 
agreement <- rater1 == rater2
n_categories <- 3
n_ratings <- 15

# The JAGS model definition, should work in WinBugs with minimal modification
kohen_model_string <- "model {
  kappa <- (p_agreement - chance_agreement) / (1 - chance_agreement)
  chance_agreement <- sum(p1 * p2)

  for(i in 1:n_ratings) {
    rater1[i] ~ dcat(p1)
    rater2[i] ~ dcat(p2)
    agreement[i] ~ dbern(p_agreement)

  # Uniform priors on all parameters
  p1 ~ ddirch(alpha)
  p2 ~ ddirch(alpha)
  p_agreement ~ dbeta(1, 1)
  for(cat_i in 1:n_categories) {
    alpha[cat_i] <- 1

# Running the model
kohen_model <- jags.model(file = textConnection(kohen_model_string),
                 data = list(rater1 = rater1, rater2 = rater2,
                   agreement = agreement, n_categories = n_categories,
                   n_ratings = n_ratings),
                 n.chains= 1, n.adapt= 1000)

update(kohen_model, 10000)
mcmc_samples <- coda.samples(kohen_model, variable.names="kappa", n.iter=20000)

The plot below shows a density plot of the MCMC samples from the posterior distribution of Kappa.

Posterior Kappa density

Using the MCMC samples we can now use the median value as an estimate of Kappa and use the 2.5% and 97.5% quantiles as a 95 % confidence/credible interval.

##      2.5%        25%        50%        75%      97.5% 
## 0.01688361 0.26103573 0.38753814 0.50757431 0.70288890 

Compare this with the "classical" estimates calculated according to Fleiss, Cohen and Everitt:

cohen.kappa(cbind(rater1, rater2), alpha=0.05)
##                  lower estimate upper
## unweighted kappa  0.041     0.40  0.76

Personally I would prefer the Bayesian confidence interval over the classical confidence interval, especially since I believe the Bayesian confidence interval have better small sample properties. A common concern people tend to have with Bayesian analyses is that you have to specify prior beliefs regarding the distributions of the parameters. Fortunately, in this case, it is easy to construct "objective" priors by simply putting uniform distributions over all the parameters. This should make the outcome of the Bayesian model very similar to a "classical" calculation of the Kappa coefficient.


Sanjib Basu, Mousumi Banerjee and Ananda Sen (2000). Bayesian Inference for Kappa from Single and Multiple Studies. Biometrics, Vol. 56, No. 2 (Jun., 2000), pp. 577-582

share|improve this answer
Do you know if there's an extension of this for more than two raters? – Fomite 17 hours ago

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.