6
$\begingroup$

I understand the formula behind the Kappa statistic value and how to calculate the O and E value from a confusion matrix.

My question is what is the intuition behind this measure? Why does it work so well for a given data set and why is it a good benchmark measure used to compare performance of different classifiers over different datas

$\endgroup$
5
$\begingroup$

The way it is usually described is the amount of agreement correct by the agreement expected by chance. However, it technically isn't corrected by chance but instead reports if the agreement is greater than by chance. Although the Kappa statistic is widely used, I believe it is most generally applied to predictive models built from unbalanced data (i.e. class distributions not equivalent). You say you understand the mathematics behind the statistic so I will not discuss it here. Let's take a look at an example using R.

# build a starting dataframe, will change shortly
df <- data.frame(act = rep(LETTERS[1:2], each=10), pred = rep(sample(LETTERS[1:2], 20, replace=T)))

# create working frequency table
tab <- table(df)

# A balanced dataset
tab[1,1] <- 45
tab[1,2] <- 5
tab[2,1] <- 5
tab[2,2] <- 45

#truncated output
caret::confusionMatrix(tab)
> caret::confusionMatrix(tab)
Confusion Matrix and Statistics

   pred
act  A  B
  A 45  5
  B  5 45

 Accuracy : 0.9            
 ...                              
 Kappa : 0.8            
 ...    

# An unbalanced datasest
tab[1,1] <- 85
tab[1,2] <- 5
tab[2,1] <- 5
tab[2,2] <- 5

caret::confusionMatrix(tab)
> caret::confusionMatrix(tab)
Confusion Matrix and Statistics

   pred
act  A  B
  A 85  5
  B  5  5

 Accuracy : 0.9            
 ...                              
 Kappa : 0.444            
 ...    

As you can see, you can have the exact same accuracy with two different datasets but very different Kappa. The idea herein being, with unbalanced data, there is a higher chance you will randomly classify the less common group so this should be accounted for in your evaluation of the model. If you dataset is balanced, you have much more flexibility with your performance metrics. It is important to keep in mind that Kappa is not always the best metric. Some pros and cons of Kappa are reported here. You should always keep in mind other methods like the AUROC (Area under the Receiver Operator Curve) and make the best informed decision for your data.

$\endgroup$
  • 1
    $\begingroup$ Thanks. Can you please explain what this statement means " The idea herein being, with unbalanced data, there is a higher chance you will randomly classify the less common group".. Does that mean there is a higher chance that the minority class will be misclassified as compared to the majority class? $\endgroup$ – London guy Nov 14 '14 at 15:51
  • 2
    $\begingroup$ Yes, many models pay more attention to the more common class as opposed to the rare class. As such, there is often more misclassification. This is clearly a bad thing if your rare group is very important (e.g. cancer presence). $\endgroup$ – cdeterman Nov 14 '14 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.