Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

Cohen's kappa is a measurement of the degree to which 2 rating systems (typically people making ratings) agree. It adjusts for the probability that the raters would agree by chance alone if they were completely independent.

The sampling distribution of kappa is known, so the measure can be used as a statistical test of agreement.

The standard calculation of kappa is: $$ \kappa=\frac{p(\text{agreement})-p(\text{chance agreement})}{1-p(\text{chance agreement})} $$ There are also weighted versions of kappa, and other related measures of inter-rater agreement.

history | excerpt history