Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

What inter-rate reliability test is best for continuous data? I am doing a study with one variable with continuous data, now the measurement involves measurements done by two people. I would wish to do inter-rater reliability test for the data, so far I have collected a few samples and a sample data I have given below, what test would I use?

Rater A 23.1 22.0 21.8 24.1 20.2 22.1 23.8

Rater B 23.0 21.5 22.0 23.9 19.8 21.9 22.9

share|improve this question
SORRY THE DATA IS – Edwin Feb 28 '11 at 17:01
1  
Also, there really is no single "best" test for reliability. Most inter-rater reliability statistics I've encountered are for nominal, ordinal, interval or ratio level data. It's possible that with continuous data, something more like correlation or rank correlation statistics may be more useful. I'd encourage you to look into what sorts of techniques are common in your particular field and/or to say more about the scale involved if you'd like a more informative answer. – ashaw Feb 28 '11 at 17:25
@ashaw No, you can't use simple correlation-based measures since they are invariant by shifting the mean of one of the two measures, or adding a constant amount to one of them, which obviously lead to decreased reliability (overall, the two raters disagree to a larger extent) but that would not be reflected in such an association measure. – chl♦ Feb 28 '11 at 19:27
@chl - Definitely, I wouldn't expect simple correlation to actually capture reliability at all. In my head, the emphasis was on the "something more like" part of that sentence. Should be clearer from these comments at least... – ashaw Feb 28 '11 at 20:12

1 Answer

up vote 1 down vote

I'd suggest you plot the difference against the mean then quantify things using the mean difference and the standard deviation of the difference. Seven samples is rather few though.

share|improve this answer
Thank you so much for that!! Do you mean the Bald-Altman plot. So the Kappa score is not applicable in the continuous data?? am yet collecting more data( my sample size is 146, but i intend to test 15 for inter-rater reliability) – Edwin Feb 28 '11 at 19:19
@Edwin @onestop's response is fine, and you can complete this graphical/numerical approach by computing an Intraclass correlation, for example. Several earlier posts dealt with those topics (search around the two tags you associate to your question). I also gave a +1 to your question, so that you can vote up @onestop's response as you seem to consider it being helpful. – chl♦ Feb 28 '11 at 19:31
@Chl..Thanks a lot for that. I just want to get it clear, so the Bland-Altman plot will be the solution... – Edwin Feb 28 '11 at 19:40
@Edwin Happy to know that you find B-A plot a good compromise. In this case, you might just upvote @onestop's response and wait few days to see if no other response come up before accepting it. – chl♦ Feb 28 '11 at 21:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.