How can I calculate inter-rater reliability for continuous measures that are not normally distributed?

Question

My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data set is comprised of proportional data (bounded from 0 to 1) that is not count data.

For a data set of n images, where each one is assessed by the same set of r raters, what is the best way of assessing interrater reliability? Some form of kappa seems inappropriate, since p is a continuous variable. Intraclass correlation makes more sense, but requires an assumption of normality for p, which is not the case here. Is this one of the few cases where arcsine transformation is actually appropriate?

Answer 1

As a quick answer, perhaps Krippendorff's alpha is more useful and robust than kappa?

IRR makes sense to me, however, for discrete raters and discrete codes, not "continuous codes." For example, perhaps you want to know the interrater reliability of the underlying "coding process" of a firm's digital thermometers. The thermometers will code temperature p discretely by 1 degree Celsius units, or 0.1, or 0.0001 but not continuously. Then it is a question of judgment whether poor interrater reliability among digital thermometers is worrisome at 0.0001 C or not, but that does not change the fact that it is poor.

To measure IRR of your "coding process," you should analyze data as coders produced it (I assume 0.01 but it could be wildly diverse) under your actual "coding process" rather than make ex-post adjustments. To achieve a higher (desirable) IRR, your "coding process" which currently asks raters to "code the proportion" may need to ask raters to "code the proportion by hundredths [deciles, quintiles, halves]."