ICC Agreement with Repeated Measures

Question

I've got measurements from two different types of sites, lets call them "Source" and "Sensor", with a proportion measure for each ranging from 0 to 1, like so:

Source, Sensor
0.25, 0.34
0.05, 0.72
0.38, 0.26

There's some interest in finding out if Sensor could be used as a proxy for Source, as Source is much more difficult to collect samples in. The two measurements are paired, so it makes sense to me to just use an ICC agreement statistic.

Here's the rub: In addition to multiple samples of each type of site, there's potentially multiple visits to the actual sites. So for example, the data actually looks like:

SourceID, Source, Sensor
A, 0.25, 0.34
A, 0.17, 0.28
A, 0.19, 0.45
B, 0.06, 0.72
C, 0.38, 0.26
D, 0.89, 0.97
D, 0.73, 0.85
E, 0.17, 0.19

I'd like to leverage those multiple visits, but of course now all the observations aren't independent, and there are some repeated measures. Is there a way to handle ICC with repeated measures?

Answer 1

You can do this using the tools from generalizability theory. You can think of these as extensions of the ICC that allow for multiple sources of variation (e.g., source of measurement and repeated measures).

Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.

Cronbach, L.J., Nageswari, R., & Gleser, G.C. (1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16, 137-163.

Shavelson, R.J., & Webb, N.M. (1991). Generalizability Theory: A Primer. Thousand Oaks, CA: Sage.