After trying to understand different implementations of intraclass correlation (ICC), inter-rater reliability, and bland-altman statistics, I'm not quite sure what is the right choice here, and would appreciate being set right.

I am analysing data from an experiment with human participants using two commonly employed, continuous measurements from the literature. For each participant, there were 6 experimental conditions, with 1 observation per condition. I use the same observations to produce the two measurements. Hence, I now have 6 observations using one measurement, and 6 observations using the other measurement.

My research has clinical implications and it's important to understand whether the two measurements capture similar information on an individual to individual level. I don't need to know that within-participant observations are correlated - for example, I already know that the absolute magnitude of one measurement tends to be greater than that of the other. Rather, I want to know that specific responses to experimental conditions would be ranked the same way by the two different measurements.

Put otherwise, my goal is to quantify how similar the pattern of results, within participant, is across the two measurements. I think this would constitute an estimate of agreement. For instance, imagine my first measurement tells me that the ranking for some participant is [B, D, A, C, E, F]. The alternative measurement says the ranking is [B, E, A, C, D, F]. I would want to quantify that this outcome is more similar than if the alternative measure had instead given the ranking [A, F, E, C, D, B].

Originally, I had thought I could calculate the Spearman correlation, within each participant, between the two measurements, and then take the group mean correlation statistic as an estimate. But I couldn't find an example of this approach and wonder if there's some obvious reason not to.

My question is quite similar to this one, which did receive an answer, but it sounds as though the asker needed to provide additional details, and that the method they proposed using wouldn't be appropriate for repeat measures.

Other related, but inconclusive discussion:

How to test if two methods of measurement are consistent?

Comparison of two measurement methods - determining measurement reliability



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.