1
vote
0answers
373 views

Which correlation analysis is appropriate for binary and ordinal variables?

Violin players were asked to rate 5 violins from least to most rich on a [0 1], 0.05-increment scale, ...
2
votes
0answers
121 views

Testing graded response model thresholds for significance?

I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
7
votes
1answer
282 views

What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
7
votes
1answer
871 views

How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?

Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
3
votes
2answers
408 views

Combining ratings from multiple raters of different accuracy

I have a data set of J raters who each give ratings to I objects (on say a scale of 1 to 5). My goal would be to construct some kind of overall rating for each i based on all of the raters' scores. ...
3
votes
2answers
338 views

The best measure of reliability for interval data between 0 and 1 [closed]

I have 6 sets of interval data each of which between 0 and 1. Each set, calculated by a computer program, is related to the degree of similarity between some sounds (pairwise). What do you think in ...
15
votes
2answers
10k views

Inter-rater reliability for ordinal or interval data

Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...