What is the intuition behind the Kappa statistical value in classification

Question

I understand the formula behind the Kappa statistic value and how to calculate the O and E value from a confusion matrix.

My question is what is the intuition behind this measure? Why does it work so well for a given data set and why is it a good benchmark measure used to compare performance of different classifiers over different datas

Answer 1

The way it is usually described is the amount of agreement correct by the agreement expected by chance. However, it technically isn't corrected by chance but instead reports if the agreement is greater than by chance. Although the Kappa statistic is widely used, I believe it is most generally applied to predictive models built from unbalanced data (i.e. class distributions not equivalent). You say you understand the mathematics behind the statistic so I will not discuss it here. Let's take a look at an example using R.

# build a starting dataframe, will change shortly
df <- data.frame(act = rep(LETTERS[1:2], each=10), pred = rep(sample(LETTERS[1:2], 20, replace=T)))

# create working frequency table
tab <- table(df)

# A balanced dataset
tab[1,1] <- 45
tab[1,2] <- 5
tab[2,1] <- 5
tab[2,2] <- 45

#truncated output
caret::confusionMatrix(tab)
> caret::confusionMatrix(tab)
Confusion Matrix and Statistics

   pred
act  A  B
  A 45  5
  B  5 45

 Accuracy : 0.9            
 ...                              
 Kappa : 0.8            
 ...    

# An unbalanced datasest
tab[1,1] <- 85
tab[1,2] <- 5
tab[2,1] <- 5
tab[2,2] <- 5

caret::confusionMatrix(tab)
> caret::confusionMatrix(tab)
Confusion Matrix and Statistics

   pred
act  A  B
  A 85  5
  B  5  5

 Accuracy : 0.9            
 ...                              
 Kappa : 0.444            
 ...    

As you can see, you can have the exact same accuracy with two different datasets but very different Kappa. The idea herein being, with unbalanced data, there is a higher chance you will randomly classify the less common group so this should be accounted for in your evaluation of the model. If you dataset is balanced, you have much more flexibility with your performance metrics. It is important to keep in mind that Kappa is not always the best metric. Some pros and cons of Kappa are reported here. You should always keep in mind other methods like the AUROC (Area under the Receiver Operator Curve) and make the best informed decision for your data.