5
$\begingroup$

I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement.

Can anyone advise on how to calculate a kappa statistic that is adjusted to account for prevalence of the object of measurement?

$\endgroup$
  • $\begingroup$ @timbp has a good answer here. I just wanted to provide the following link to a page that has a very nice, comprehensive, introductory overview of all issues related to inter-rater agreement. You may want to read through it. $\endgroup$ – gung Dec 12 '11 at 6:43
10
$\begingroup$

Kappa is regarded as a measure of chance-adjusted agreement, calculated as $\frac{{{p_{obs}} - {p_{exp}}}}{{1 - {p_{exp}}}}$ where ${p_{obs}} = \sum\limits_{i = 1}^k {{p_{ii}}} $ and ${p_{exp}} = \sum\limits_{i = 1}^k {{p_{i + }}{p_{ + i}}} $ ($p_{i+}$ and $p_{+i}$ are the marginal totals). Essentially, it is a measure of the agreement that is greater than expected by chance.

Where the prevalence of one of the categories is high, chance agreement will be high and kappa can have unexpectedly low values. To adjust for this, and for the bias of kappa (I don't have sources nearby to refresh my memory to be able to write about the bias), the prevalence and bias adjusted kappa (PABAK) can be used. This can be calculated as $\frac{{k{p_{obs}} - 1}}{{k - 1}}$ where $k$ is the number of categories.

There is some difference of opinion on whether to use PABAK. Some commentators believe the prevalence and bias effects are important in themselves so kappa should be reported, together with measures of prevalence and bias. Others like the convenience of a single number. I chose to report both PABAK and kappa when I needed to use kappa.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.