My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data set is comprised of proportional data (bounded from 0 to 1) that is not count data.

For a data set of n images, where each one is assessed by the same set of r raters, what is the best way of assessing interrater reliability? Some form of kappa seems inappropriate, since p is a continuous variable. Intraclass correlation makes more sense, but requires an assumption of normality for p, which is not the case here. Is this one of the few cases where arcsine transformation is actually appropriate?


As a quick answer, perhaps Krippendorff's alpha is more useful and robust than kappa?

IRR makes sense to me, however, for discrete raters and discrete codes, not "continuous codes." For example, perhaps you want to know the interrater reliability of the underlying "coding process" of a firm's digital thermometers. The thermometers will code temperature p discretely by 1 degree Celsius units, or 0.1, or 0.0001 but not continuously. Then it is a question of judgment whether poor interrater reliability among digital thermometers is worrisome at 0.0001 C or not, but that does not change the fact that it is poor.

To measure IRR of your "coding process," you should analyze data as coders produced it (I assume 0.01 but it could be wildly diverse) under your actual "coding process" rather than make ex-post adjustments. To achieve a higher (desirable) IRR, your "coding process" which currently asks raters to "code the proportion" may need to ask raters to "code the proportion by hundredths [deciles, quintiles, halves]."

  • $\begingroup$ Thanks for your comment and your time. I agree that IRR as formulated by the kappas or my understanding of Krippendorff's alpha makes most sense for discrete data/coding. Hence why I was looking into something like ICC, specifically formulated to deal with continuous data. $\endgroup$ – user3246592 Feb 27 '15 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.