# Adjusting kappa inter-rater agreement for prevalence

I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement.

Can anyone advise on how to calculate a kappa statistic that is adjusted to account for prevalence of the object of measurement?

• @timbp has a good answer here. I just wanted to provide the following link to a page that has a very nice, comprehensive, introductory overview of all issues related to inter-rater agreement. You may want to read through it. – gung Dec 12 '11 at 6:43

Kappa is regarded as a measure of chance-adjusted agreement, calculated as $\frac{{{p_{obs}} - {p_{exp}}}}{{1 - {p_{exp}}}}$ where ${p_{obs}} = \sum\limits_{i = 1}^k {{p_{ii}}}$ and ${p_{exp}} = \sum\limits_{i = 1}^k {{p_{i + }}{p_{ + i}}}$ ($p_{i+}$ and $p_{+i}$ are the marginal totals). Essentially, it is a measure of the agreement that is greater than expected by chance.
Where the prevalence of one of the categories is high, chance agreement will be high and kappa can have unexpectedly low values. To adjust for this, and for the bias of kappa (I don't have sources nearby to refresh my memory to be able to write about the bias), the prevalence and bias adjusted kappa (PABAK) can be used. This can be calculated as $\frac{{k{p_{obs}} - 1}}{{k - 1}}$ where $k$ is the number of categories.