The best measure of reliability for interval data between 0 and 1 [closed]

I have 6 sets of interval data each of which between 0 and 1. Each set, calculated by a computer program, is related to the degree of similarity between some sounds (pairwise). What do you think in the best inter-rater reliability measure I can use to see how close the 6 judges are? If I want to explain the data in each set, it can be: 0.98, 0.01, 0.5, ... which shows 'sound1' and 'sound2' are very similar (0.98), 'sound1' and 'sound3' are much different (0.01) and so on. Thank you so much.

-
@Shadi How many discrete values do you have? Are they really to be considered to be ordinal, if they lie between 0 and 1? –  chl Oct 19 '10 at 15:42
They are continuous, every float value between 0 and 1. Is there any incoherence between ordinal data and [0 1] interval? Thanks. –  shadi Oct 19 '10 at 15:49
For me this would rather be considered an interval scale: en.wikipedia.org/wiki/Interval_scale#Interval_scale –  Henrik Oct 19 '10 at 15:54
Yeah, I believe you are right. So you mean I cannot use ordinal measures for interval data, right? I corrected my question. I am looking forward to your guidances. Thanks. –  shadi Oct 19 '10 at 15:59
@Shadi Following my latest comment, I asked for closing this question so that you can reformulate a new one by adding precision on your design, especially the fact that you actually have 6 similarity matrices instead of 6 series of measurement. This way, others may provide useful insights into this question. You can still link to this question, but I really feel it call for a new thread with your added clarifications so that everyone can contribute. –  chl Oct 20 '10 at 17:34

closed as not a real question by mbq♦Oct 20 '10 at 17:16

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

Referring to your comments to @Henrik, I'm inclined to think that you rather have continuous measurements on a set of objects (here, your similarity measure) for 6 raters. You can compute an intraclass correlation coefficient, as described here Reliability in Elicitation Exercise. It will provide you with a measure of agreement (or concordance) between all 6 judges wrt. assessments they made, or more precisely the part of variance that is explained by between-rater variance. There's a working R script in appendix.

Note that this assumes that your measures are considered as real valued measurement (I refer to @onestop's comment), not really proportions of similarity or whatever between your paired sounds. I don't know of a specific version of the ICC for % or values bounded on an interval, only for binary or ranked data.

Update:

• There are many other online ressources on the ICC; I think David Howell provides a gentle and well illustrated introduction to it. Its discussion generalize to k-sample (judges/raters) without any difficulty I think, or see this chapter from Sea and Fortna on Psychometric Methods. What you have to think to is mainly whether you want to consider your raters as an unique set of observers, not necessarily representative of all the raters that would have assess your object of measurement (this is called a fixed effect), or as a random sample of raters sampled from a larger (hypothetical) population of potential raters: in the former case, this corresponds to a one-way anova or a consistency ICC, in the latter case we talk about an agreement ICC.

• A colleague of mine successfully used Kevin Brownhill's script (from Matlab Central file exchange). The ICC you are interested in is then cse=3 (if you consider that your raters are not representative of a more general population of raters).

-
If I want to explain the data in each set, it can be: 0.98, 0.01, 0.5, ... which shows 'sound1' and 'sound2' are very similar (0.98), 'sound1' and 'sound3' are much different (0.01) and so on. In this case, do you believe I can use ICC or Pearson is better? –  shadi Oct 19 '10 at 16:59
@Shadi Did you read the thread I pointed to? You cannot use correlation-based criteria. So, I would better advice you to rely on the ICC unless someone has a better idea to cope with bounded values. For me it's not a problem as you're likely to end-up with similar results than with any other most complicated method. Still I agree with others than it makes sense not to use ANOVA or mixed-effects models with inappropriate link function in certain cases. –  chl Oct 19 '10 at 17:00
Yes, I have studied them. Thank you. Actually I am completely new to this concept. I did not understand almost 80% of the thread. I wanted to use ICC in Matlab, but at first I should know the meaning of 'type', 'alpha' and 'r0' to know what value is the best for my purpose. Do you know any quick way to get some information? Thanks for your guidance. –  shadi Oct 19 '10 at 18:17
@Shadi Oups, sorry, I didn't think of language issue. I've updated my response. HTH –  chl Oct 19 '10 at 19:42
Dear chl, Thanks for your great guides. I'm not sure if I explained my data well or not. when you asked : "how many discrete data do you have", did you mean something like "partner1" and "partner2" in (uvm.edu/~dhowell/StatPages/More_Stuff/icc/icc.html)? If so, I should say just 1. I try to explain my data better. I have, for example, 100 different sounds. I want to know how similar each two sounds are. I used a program to do that for me. It generated some (4851) numbers. I also used 5 other programs to do the same thing. Now, I want to know how close the results of the 6 programs are –  shadi Oct 19 '10 at 22:43
@Shadi Yes, I understand your design; I just take as an illustration @Henrik's POV with 2 series; the same line of reasoning applies with $k$ series of measurement. –  chl Oct 19 '10 at 16:44