My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data set is comprised of proportional data (bounded from 0 to 1) that is not count data.
For a data set of n images, where each one is assessed by the same set of r raters, what is the best way of assessing interrater reliability? Some form of kappa seems inappropriate, since p is a continuous variable. Intraclass correlation makes more sense, but requires an assumption of normality for p, which is not the case here. Is this one of the few cases where arcsine transformation is actually appropriate?