Inter-rater reliability refers to the agreement (often measured by correlation) between two raters when rating a set of items. It is one part of properly developing a measure.

learn more… | top users | synonyms

0
votes
0answers
11 views

Statistical formula for intra-rater reliability

I designed an assessment rubric for a class assignment and wish to calculate my own intra-rater reliability. My goal is to use the rubric to evaluate the same ordinal data at two different points in ...
0
votes
0answers
41 views

Inter-rater reliability: multiple raters and multiple replications

This is really a two part question. I have been asked to assess inter-rater agreement on a study in which 4 raters were asked to rate microarrays from patient samples. Ratings were based on a 5 point ...
1
vote
1answer
44 views

How to summarize measurement variability for repeated measures?

I have a data set where different measures are taken repeatedly (3) times for multiple samples, like so: A B C X ... Y ... So, for people X, Y, etc I have ...
0
votes
0answers
15 views

How can we measure between-experimenter reliability for experimental data?

Description of my situation: I will run an experiment in a few weeks with a total sample size of N = 70. Ideally I would prefer to have only one experimenter to collect all the data but I have to do ...
0
votes
0answers
43 views

Intraclass correlation (ICC) per item instead of across items

I am using the intraclass correlation coefficiant to determine inter-rater reliability for a set of pictures that were rated by several raters on a scale from 1-9. I am using the two-way mixed model ...
0
votes
0answers
29 views

consensus scoring model that maximizes ranking agreement with real data

Consider the following dataset: OBJECT: 1 2 3 4 ... 15 SFA: 3.5 2.2 4.0 3.6 ... 2.8 rankingA: 3 9 1 2 ... 7 SFB: 3.1 3.1 3.7 3.4 ... 2.7 rankingB: 4.5 4.5 2 3 ... 9 SFC: 3.3 3.0 3.7 3.5 ... 2.1 ...
0
votes
0answers
37 views

Optimum linear combination of predictors that maximizes a quantity

I use 48 energy functions to score protein-ligand interactions. I have a dataset of protein-ligands for which I know the experimental binding energy, so I can compare it with the score assigned from ...
0
votes
0answers
35 views

Finding the optimum combination of raters that maximizes a quantity

I have scored the binding affinity of a set of small-molecules relative to protein receptor using multiple scoring function (consider them as "raters"). I also have the experimental binding affinities ...
1
vote
0answers
82 views

Inter-rater agreement with multiple raters

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
1
vote
0answers
94 views

Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
3
votes
0answers
55 views

What is the analysis techniques for comparing two sets of ratings

Say, I have collected 50 comments. For each comment, I am planning to get 10 ratings from non-expert raters for the following criteria: "interesting vs. not interesting". Based on the plurality of ...
0
votes
0answers
183 views

Sample size for multiple rater inter-rater reliability study

How do I calculate the minimum overall sample size required for a study using multiple raters (n=5) assessing inter-rater reliability using Fleiss generalized kappa for binary variables, Kendall's W ...
2
votes
1answer
510 views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
3
votes
2answers
3k views

Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS. Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important). All ...
1
vote
0answers
56 views

Is Fleiss's $\kappa$ as conservative as Cohen's $\kappa$?

I know Cohen's $\kappa$ for inter-rater reliability is often regarded as a conservative estimation--e.g., Strijbos & al. below, who say The higher the number of categories, the smaller the ...
1
vote
1answer
641 views

Kendall's coefficient of concordance (W) for ratings with a lot of ties

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As ...
1
vote
0answers
68 views

Looking for agreement amongst respondents

I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of ...
0
votes
0answers
84 views

Intra-rater reliability (measurement error) with linear data - 1 rater, video performances scored 3 times

Background: I am an occupational therapist doing a PhD in clinical left hemispheric stroke assessment and intra- individual (patient) variability. I'm in my feasibility phase and I need to calculate ...
3
votes
0answers
226 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
0
votes
0answers
128 views

interrater reliability for multiple variables

i and another coder want to assess our agreement. our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
2
votes
1answer
30 views

Macro and Micro Inter-Coder/Rater Agreement

I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in ...
0
votes
0answers
104 views

How to assess Inter-rater reliability of frequency data with two raters using SPSS?

I am doing an observational study on the frequency (number of times) a particular person enters a shop. This study is being carried out by two different observers. I now want to do an inter-rater ...
5
votes
1answer
99 views

Calculating inter rater reliability where raters and ratees only partially overlap

I'm new to inter rater reliability calculations. I have 5 developers and 6 raters. The raters ranked the first 3 of these developers based on some criteria like the following example: For ...
2
votes
1answer
117 views

How to assess relationship between responses to multiple likert items between two raters when there are multiple rater pairs?

I have two data sets in which the same Likert scale questions were answered by two different responders. I want to calculate which level they are correlated. My data set is like ...
4
votes
1answer
138 views

How to evaluate differences of multiple Likert scale ratings of multiple items within groups?

I have data where different reviewers rated how important multiple topics are on a scale of 1 to 5 (Likert scale). Each of the topics is assigned to a group. I want to see if there any differences in ...
1
vote
0answers
97 views

What is the proper test for inter-rater reliability (three raters, ordinal data)

I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal ...
1
vote
0answers
300 views

Which correlation analysis is appropriate for binary and ordinal variables?

Violin players were asked to rate 5 violins from least to most rich on a [0 1], 0.05-increment scale, ...
7
votes
2answers
292 views

Handling poor inter-rater reliability while minimizing the loss of data

Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
2
votes
0answers
82 views

Assessing the ability of two quantitative methods measuring the same thing

Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
1
vote
1answer
48 views

Test for treatment change

I have a study where the goal is to see if a new diagnostic tool provides any addition to the old procedure. A set of patients have been diagnosed and classified into 7 categories, each corresponding ...
2
votes
1answer
188 views

Basic Inter-rater reliability and type of data

I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability. Research question: Functional Movement Screening: How reliable are we in our ...
1
vote
2answers
390 views

Determining the number of raters for inter-rater reliability

I intend to ask a number of experts to match five out of a possible ten team roles (recorder, chairman and so on) to five distinct stages of a project. I cannot seem to find any source for ...
1
vote
0answers
121 views

Testing marginal homogeniety with multivariate outcomes and matched pairs

Suppose I have 20 people matched in 10 pairs. Suppose two doctors are assigned at random to diagnose one person in each pair, so in each pair each person is diagnosed by a different doctor. ...
3
votes
2answers
393 views

Inconsistency of findings between Kappa and ICC for IRR study

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
2
votes
2answers
77 views

Difference in proportions but one of them is 0

What is the best way to test for difference of proportions when of them is 0% (or 100%) ? edit: I have two raters, who give a score of either 1 or 0. One of them gave a 0 in every single case. Now I ...
2
votes
0answers
103 views

Testing graded response model thresholds for significance?

I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
1
vote
0answers
243 views

How should I interpret the Kappa Fleiss test?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
2
votes
0answers
95 views

Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?

For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
1
vote
1answer
163 views

Need normal distribution for kappa Fleiss and ICC (2, 1)?

I'm analysing a study of 4 raters and 34 subjects. I'm looking at the inter rater reliability. For the ordinal results, I did a weighted kappa of Fleiss and for the other continuous results I ...
2
votes
0answers
56 views

Relative scoring system without much effect from outliers

I am trying to assign scores to phone's specifications to calculate a final rating score for each phone. However, when I was scoring based on megapixels. I found that one of the phones 'A' has a 41MP ...
2
votes
0answers
66 views

What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?

I found a question that is related here, but it doesn't really goes on what I want to know. I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...
7
votes
1answer
272 views

What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
20
votes
2answers
556 views

Is Joel Spolsky's “Hunting of the Snark” post valid statistical content analysis?

If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He ...
1
vote
2answers
455 views

How to measure agreement between a set of raters?

I am running an experiment where I have a finite set of raters and a finite set of items, and raters have to provide their subjective judgment about each item. The goal is to measure the importance of ...
7
votes
1answer
836 views

How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?

Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
4
votes
1answer
629 views

Interrater reliability in SPSS

I am trying to calculate interrater reliability in SPSS for both pre and post test of the same measure that is administered as part of a prison intake program. The measure has 20 items and then a ...
4
votes
3answers
283 views

Inter-rater statistic for skewed rankings

I have several sets of 10 raters which I want to compare. Each rater can cast only Yes or No vote, however this decision is skewed and the Yes votes make only about 10% of all votes (and this is ...
2
votes
2answers
243 views

Inter-rater reliability of time measurements

I am having some trouble trying to decide what measure of inter-rater reliability to use in a study. Part of a larger study involves accurately determining when participants began (onset) and stopped ...
1
vote
2answers
206 views

Measuring inter-subject agreement in user ratings

In my system I have users rating objects (e.g. films) from 1 to 5 stars. The mean rating clearly tells something about the overall opinion on an object, but I'd like to use a more precise measure of ...
3
votes
2answers
352 views

Combining ratings from multiple raters of different accuracy

I have a data set of J raters who each give ratings to I objects (on say a scale of 1 to 5). My goal would be to construct some kind of overall rating for each i based on all of the raters' scores. ...