Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement.

Can anyone advise on how to calculate a kappa statistic that is adjusted to account for prevalence of the object of measurement?

share|improve this question

2 Answers 2

Kappa is regarded as a measure of chance-adjusted agreement, calculated as $\frac{{{p_{obs}} - {p_{exp}}}}{{1 - {p_{exp}}}}$ where ${p_{obs}} = \sum\limits_{i = 1}^k {{p_{ii}}} $ and ${p_{exp}} = \sum\limits_{i = 1}^k {{p_{i + }}{p_{ + i}}} $ ($p_{i+}$ and $p_{+i}$ are the marginal totals). Essentially, it is a measure of the agreement that is greater than expected by chance.

Where the prevalence of one of the categories is high, chance agreement will be high and kappa can have unexpectedly low values. To adjust for this, and for the bias of kappa (I don't have sources nearby to refresh my memory to be able to write about the bias), the prevalence and bias adjusted kappa (PABAK) can be used. This can be calculated as $\frac{{k{p_{obs}} - 1}}{{k - 1}}$ where $k$ is the number of categories.

There is some difference of opinion on whether to use PABAK. Some commentators believe the prevalence and bias effects are important in themselves so kappa should be reported, together with measures of prevalence and bias. Others like the convenience of a single number. I chose to report both PABAK and kappa when I needed to use kappa.

share|improve this answer

@timbp has a good answer here. I just wanted to provide the following link to a page that has a very nice, comprehensive, introductory overview of all issues related to inter-rater agreement. You may want to read through it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.