Inter-rater reliability refers to the agreement (often measured by correlation) between two raters when rating a set of items. It is one part of properly developing a measure.
0
votes
0answers
25 views
interrater reliability for multiple variables
i and another coder want to assess our agreement.
our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
1
vote
1answer
9 views
Macro and Micro Inter-Coder/Rater Agreement
I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in
...
0
votes
0answers
36 views
How to assess Inter-rater reliability of frequency data with two raters using SPSS?
I am doing an observational study on the frequency (number of times) a particular person enters a shop. This study is being carried out by two different observers. I now want to do an inter-rater ...
5
votes
1answer
44 views
Calculating inter rater reliability where raters and ratees only partially overlap
I'm new to inter rater reliability calculations.
I have 5 developers and 6 raters. The raters ranked the first 3 of these developers based on some criteria like the following example:
For ...
3
votes
0answers
47 views
How to evaluate differences of multiple likert scale ratings of multiple items within groups?
I have data where different reviewers rated how important multiple topics are on a scale of 1 to 5 (likert scale). Each of the topics is assigned to a group. I want to see if there any differences in ...
1
vote
0answers
63 views
What is the proper test for inter-rater reliability (three raters, ordinal data)
I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal ...
1
vote
0answers
184 views
Which correlation analysis is appropriate for binary and ordinal variables?
Violin players were asked to rate 5 violins from least to most rich on a [0 1], 0.05-increment scale, ...
7
votes
2answers
208 views
Handling poor inter-rater reliability while minimizing the loss of data
Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
2
votes
0answers
50 views
Assessing the ability of two quantitative methods measuring the same thing
Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
1
vote
1answer
45 views
Test for treatment change
I have a study where the goal is to see if a new diagnostic tool provides any addition to the old procedure. A set of patients have been diagnosed and classified into 7 categories, each corresponding ...
2
votes
1answer
88 views
Basic Inter-rater reliability and type of data
I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability.
Research question: Functional Movement Screening: How reliable are we in our ...
0
votes
1answer
173 views
Determining the number of raters for inter-rater reliability
I intend to ask a number of experts to match five out of a possible ten team roles (recorder, chairman and so on) to five distinct stages of a project.
I cannot seem to find any source for ...
1
vote
0answers
51 views
Testing marginal homogeniety with multivariate outcomes and matched pairs
Suppose I have 20 people matched in 10 pairs.
Suppose two doctors are assigned at random to diagnose one person in each pair, so in each pair each person is diagnosed by a different doctor.
...
3
votes
2answers
173 views
Inconsistency of findings between Kappa and ICC for IRR study
I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects:
...
2
votes
2answers
72 views
Difference in proportions but one of them is 0
What is the best way to test for difference of proportions when of them is 0% (or 100%) ?
edit: I have two raters, who give a score of either 1 or 0. One of them gave a 0 in every single case. Now I ...
2
votes
0answers
72 views
Testing graded response model thresholds for significance?
I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
0
votes
0answers
156 views
How should I interpret the Kappa Fleiss test?
I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
2
votes
0answers
71 views
Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?
For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
1
vote
1answer
116 views
Need normal distribution for kappa Fleiss and ICC (2, 1)?
I'm analysing a study of 4 raters and 34 subjects. I'm looking at the inter rater reliability.
For the ordinal results, I did a weighted kappa of Fleiss and for the other continuous results I ...
2
votes
0answers
49 views
Relative scoring system without much effect from outliers
I am trying to assign scores to phone's specifications to calculate a final rating score for each phone. However, when I was scoring based on megapixels. I found that one of the phones 'A' has a 41MP ...
2
votes
0answers
51 views
What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?
I found a question that is related here, but it doesn't really goes on what I want to know.
I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...
7
votes
1answer
208 views
What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?
In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
19
votes
2answers
480 views
Is Joel Spolsky's “Hunting of the Snark” post valid statistical content analysis?
If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He ...
0
votes
2answers
206 views
How to measure agreement between a set of raters?
I am running an experiment where I have a finite set of raters and a finite set of items, and raters have to provide their subjective judgment about each item. The goal is to measure the importance of ...
7
votes
1answer
637 views
How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?
Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
4
votes
1answer
437 views
Interrater reliability in SPSS
I am trying to calculate interrater reliability in SPSS for both pre and post test of the same measure that is administered as part of a prison intake program. The measure has 20 items and then a ...
3
votes
3answers
198 views
Inter-rater statistic for skewed rankings
I have several sets of 10 raters which I want to compare.
Each rater can cast only Yes or No vote, however this decision is skewed and the Yes votes make only about 10% of all votes (and this is ...
2
votes
2answers
186 views
Inter-rater reliability of time measurements
I am having some trouble trying to decide what measure of inter-rater reliability to use in a study. Part of a larger study involves accurately determining when participants began (onset) and stopped ...
1
vote
2answers
172 views
Measuring inter-subject agreement in user ratings
In my system I have users rating objects (e.g. films) from 1 to 5 stars.
The mean rating clearly tells something about the overall opinion on an object, but I'd like to use a more precise measure of ...
3
votes
2answers
206 views
Combining ratings from multiple raters of different accuracy
I have a data set of J raters who each give ratings to I objects (on say a scale of 1 to 5). My goal would be to construct some kind of overall rating for each i based on all of the raters' scores.
...
2
votes
1answer
182 views
Advantages and limitations of Gwet’s AC1 statistic and PABAK
I am looking for alternatives to the kappa to assess inter-rater agreement. I've come across two hopefulls: Gwet’s AC1 statistic and PABAK. I wondered what the advantages and disadvantages of each ...
1
vote
0answers
263 views
How to analyse linked 2x2 tables from multiple raters?
Comment: heavily edited, both to clarify and to fix a mistake. First 7 comments below refer to the previous version.
I have an experiment where 3 raters go over the same data 2 times each, trying to ...
2
votes
2answers
487 views
Adjusting kappa inter-rater agreement for prevalence
I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement.
Can anyone advise on how ...
3
votes
0answers
195 views
Computing inter-rater reliability in R with variable number of ratings?
Wikipedia suggests that one way to look at inter-rater reliability is to use a random effects model to compute intraclass correlation. The example of intraclass correlation talks about looking at
...
1
vote
1answer
315 views
What inter-rater reliability test is best for continous data with multiple measures?
I am doing a study with one variable with continuous data; the measurement involves measurements done by three people. I wish to do an inter-rater reliability test, e.g. with Bland-Altman analysis, ...
1
vote
0answers
108 views
IRR on two raters’ integer scores
I am writing with questions about a study I am helping with:
Two “raters” will score athletes subjectively from 0 to 3 (3>2>1>0) on each of 10 skill performances. Scores will be summed to a maximum ...
4
votes
1answer
94 views
Ordered category agreement
I have data on two processes, where the process assigns elements into ordered bins. I am interested in testing for agreement between the processes. What is the best way to do this (R code)? Here is ...
3
votes
1answer
187 views
Inter-rater reliability with many non-overlapping raters
I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk.
88 different raters provided judgments for ...
1
vote
0answers
210 views
Kendall's W for multiple objects and multiple surveys
I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios.
For example suppose for scenario A we had multiple objects and multiple raters like in the table ...
4
votes
1answer
2k views
Intra- and inter-rater reliability on the same data
I've read some of the answers here (specifically the 'Inter-rater reliability for ordinal or interval data' one). I'm still somewhat perplexed though!
I have data for 4 raters, who each rated every ...
3
votes
2answers
787 views
What to do in case of low inter-rater reliability (ICC)?
Background: Eight doctors each rated the same 54 patients on a persuasiveness measure (1-7 Likert scale). The mean score on the persuasiveness measure will eventually be the outcome measure of my ...
3
votes
2answers
168 views
Specification of a random slopes model with two grouping factors
Suppose people take a 10-item exam. For each item, $k = 1,2,…,10$, exactly one rater assigns a score to exactly one person, with the constraint that no person sees the same rater twice. There are $i = ...
4
votes
2answers
972 views
Assessing and testing inter-rater agreement with kappa statistic on a set of binary and Likert items?
I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions.
Should ...
4
votes
4answers
228 views
Assessing rater bias where one rater has given one very high rating and the remainder very low ratings
What is a good statistical test to check if there is a bias in judging in a situation that there is one judge that gave extreme scores (high score for one of the contestant and very low scores on the ...
2
votes
1answer
568 views
Assumption of additivity for intra-class correlation
My question concerns the assumption of additivity for intraclass correlation. I shall first explain what I have done and then end with my questions.
I want to calculate inter-rater reliability using ...
3
votes
1answer
499 views
Interater reliability
What inter-rate reliability test is best for continuous data? I am doing a study with one variable with continuous data, now the measurement involves measurements done by two people. I would wish to ...
7
votes
2answers
323 views
Interrater reliability for events in a time series with uncertainty about event time
I have multiple independent coders who are trying to identify events in a time series -- in this case, watching video of face-to-face conversation and looking for particular nonverbal behaviors (e.g., ...
3
votes
2answers
261 views
How to “prove” that new measurement tool & process gives same result as old?
I am new to the area of statistics and I am hoping you can suggest methods I may use. Sorry if this is long but I might as well be as clear as possible on my first post :)
What I am worried most is ...
3
votes
1answer
6k views
Inter-rater reliability using Intra-class correlation with ratings for multiple objects on multiple properties
Initial Question:
I am trying to calculate inter-rater reliability. Previous researchers in this area have used intraclass correlation. SPSS has options for two-way random, mixed random and one-way ...
2
votes
1answer
220 views
Inter-rater reliability between similarity matrices
I have 6 similarity matrices reflecting the proximity of 100 pairs of sounds. Similarity indices are in the range [0,1], with higher values indicating sounds perceived as more alike. What is the most ...
