Cohen's kappa in plain English I am reading a data mining book and it mentioned the Kappa statistic as a means for evaluating the prediction performance of classifiers. However, I just can't understand this. I also checked Wikipedia but it didn't help too: https://en.wikipedia.org/wiki/Cohen's_kappa. 
How does Cohen's kappa help in evaluating the prediction performance of classifiers? What does it tell?
I understand that 100% kappa means that the classifier is in total agreement with a random classifier, but I don't understand how does this help in evaluating the performance of the classifier?
What does 40% kappa mean? Does it mean that 40% of the time, the classifier is in agreement with the random classifier? If so, what does that tell me or help me in evaluating the classifier?
 A: rbx has a great answer. However, it is a little bit verbose. Here is my summary and intuition behind the Kappa metric.

Kappa is an important measure on classifier performance, especially on imbalanced data set.
For example, in credit card fraud detection, the marginal distribution of the response variable is high skewed, that using accuracy as a measure will not be useful. In other words, for given fraud detection example, 99.9% of the transactions will be non-fraud transactions. We can have a trivial classifier that always says non-fraud to every transaction, and we will still have 99.9% of the accuracy.
On the other hand, Kappa will "fix" this problem by consider the marginal distribution of the response variable. Using Kappa, the aforementioned trivial classifier will have a very small Kappa.
In plain English, it measures how much better the classifier is, compared to guessing with the target distribution.
A: What value of Cohen's kappa is strong depends on several factors including for example, the number of categories or codes that are used affects kappa$^1$ and the probability that each code will be populated. 

"For example, given equiprobable codes and observers who are 85% accurate:

value of kappa   number of codes
0.49             2
0.60             3 
0.66             5 
0.69             10"

Now, what if we do not have equiprobable codes but have different "base rates"?
For two codes the kappa plots from Bruckner et al. would look like  
...Nonetheless (... continuing Wikipedia quote), magnitude guidelines have appeared in the literature. Perhaps the first was Landis and Koch, who characterized values 
 <0 as indicating no agreement
 0.00–0.20 as slight, 
 0.21–0.40 as fair, 
 0.41–0.60 as moderate, 
 0.61–0.80 as substantial, and 
 0.81–1 as almost perfect agreement. 

This set of guidelines is however by no means universally accepted; Landis and Koch supplied no evidence to support it, basing it instead on personal opinion. It has been noted that these guidelines may be more harmful than helpful. Fleiss's equally arbitrary guidelines characterize kappas over 
>0.75 as excellent, 
 0.40 to 0.75 as fair to good, and 
<0.40 as poor."

(end Wikipedia quote) 
For a (hard to find) upgrade of the FalliObs Windows program to account for the number of codes originally offered by Bakeman et al.$^1$ follow link to ComKappa3. The program description$^2$ relates that the standard error of kappa can be estimated, allowing  the obtained kappa  to  be tested for significance against a null distribution (Bakeman & Gottman, 1997; Fleiss, Cohen, & Everitt,  1969). For further reading for other kappa measures see ANALYSIS OF BEHAVIORAL STREAMS.
Also see Using Cohen's kappa statistic for evaluating a binary classifier for a similar question.
1 Bakeman, R.; Quera, V.; McArthur, D.; Robinson, B. F. (1997). "Detecting sequential patterns and determining their reliability with fallible observers". Psychological Methods. 2: 357–370. doi:10.1037/1082-989X.2.4.357
2 Robinson BF, Bakeman R. ComKappa: A Windows’ 95 program for calculating kappa and related statistics. Behavior Research Methods. 1998;30:731-2.
A: to answer your question (in plain english :-) ): 

How does Kappa help in evaluating the prediction performance of classifiers? What does it tell?!!

You should consider the kappa as a measure of agreement between 2 individuals such that the result can be interpreted as:
Poor agreement = 0.20 or less
Fair agreement = 0.20 to 0.40
Moderate agreement = 0.40 to 0.60
Good agreement = 0.60 to 0.80
Very good agreement = 0.80 to 1.00

