Inter-rater reliability refers to the agreement (often measured by correlation) between two raters when rating a set of items. It is one part of properly developing a measure.

learn more… | top users | synonyms

0
votes
1answer
10 views

What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
1
vote
0answers
18 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
0
votes
0answers
4 views

Which test do I run to check inter-rater agreement for multiple ordinal variables?

I'd like to check the inter-rater agreement between the six raters for the six ordinal variables scored for each of the 24 score sheets passed to each rater (6 ordinal variables per sheet, 24 sheets ...
1
vote
1answer
23 views

Do I run a one-tailed or two-tailed test when running Spearman's correlation between interval and ordinal data?

I have one interval variable on a 1-25 point interval scale, i.e. placement test scores where 25 is the highest competence score (data not normally distributed), and six different ordinal variables on ...
2
votes
0answers
14 views

Analysis of ordinal data with multiple assessments per unit… the right approach?

Suppose I have a dataset with the following information: N objects, each of which can be rated The number of ratings per object is variable (raters are unidentified), e.g., there are 30 objects each ...
1
vote
0answers
44 views

Can you test if one judge gives significantly different scores from other judges?

If I had a small number of observations (about 5) in a sample and the population standard deviation was unknown, what test could be done to see if the value of one of the observations was ...
1
vote
0answers
18 views

Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
0
votes
0answers
26 views

Measure of accuracy or measure of agreement for continuous data with true values

I have two continuous variables. Variable $X$ represents the true values (basically it is atmospheric pressure measured via a barometer). Variable $Y$ represents a way to approximate the atmospheric ...
0
votes
1answer
53 views

How will two raters score videos?

I have 260 teacher conferences to score using a rubric. How do two raters need to score these for rater agreement? Do we need three raters?
0
votes
0answers
30 views

Statistical formula for intra-rater reliability

I designed an assessment rubric for a class assignment and wish to calculate my own intra-rater reliability. My goal is to use the rubric to evaluate the same ordinal data at two different points in ...
0
votes
0answers
71 views

Inter-rater reliability: multiple raters and multiple replications

This is really a two part question. I have been asked to assess inter-rater agreement on a study in which 4 raters were asked to rate microarrays from patient samples. Ratings were based on a 5 point ...
1
vote
1answer
56 views

How to summarize measurement variability for repeated measures?

I have a data set where different measures are taken repeatedly (3) times for multiple samples, like so: A B C X ... Y ... So, for people X, Y, etc I have ...
0
votes
0answers
19 views

How can we measure between-experimenter reliability for experimental data?

Description of my situation: I will run an experiment in a few weeks with a total sample size of N = 70. Ideally I would prefer to have only one experimenter to collect all the data but I have to do ...
0
votes
0answers
74 views

Intraclass correlation (ICC) per item instead of across items

I am using the intraclass correlation coefficiant to determine inter-rater reliability for a set of pictures that were rated by several raters on a scale from 1-9. I am using the two-way mixed model ...
0
votes
0answers
36 views

consensus scoring model that maximizes ranking agreement with real data

Consider the following dataset: OBJECT: 1 2 3 4 ... 15 SFA: 3.5 2.2 4.0 3.6 ... 2.8 rankingA: 3 9 1 2 ... 7 SFB: 3.1 3.1 3.7 3.4 ... 2.7 rankingB: 4.5 4.5 2 3 ... 9 SFC: 3.3 3.0 3.7 3.5 ... 2.1 ...
0
votes
0answers
42 views

Optimum linear combination of predictors that maximizes a quantity

I use 48 energy functions to score protein-ligand interactions. I have a dataset of protein-ligands for which I know the experimental binding energy, so I can compare it with the score assigned from ...
0
votes
0answers
47 views

Finding the optimum combination of raters that maximizes a quantity

I have scored the binding affinity of a set of small-molecules relative to protein receptor using multiple scoring function (consider them as "raters"). I also have the experimental binding affinities ...
1
vote
0answers
109 views

Inter-rater agreement with multiple raters

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
1
vote
0answers
126 views

Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
3
votes
0answers
58 views

What is the analysis techniques for comparing two sets of ratings

Say, I have collected 50 comments. For each comment, I am planning to get 10 ratings from non-expert raters for the following criteria: "interesting vs. not interesting". Based on the plurality of ...
0
votes
0answers
188 views

Sample size for multiple rater inter-rater reliability study

How do I calculate the minimum overall sample size required for a study using multiple raters (n=5) assessing inter-rater reliability using Fleiss generalized kappa for binary variables, Kendall's W ...
2
votes
1answer
821 views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
3
votes
2answers
4k views

Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS. Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important). All ...
1
vote
0answers
61 views

Is Fleiss's $\kappa$ as conservative as Cohen's $\kappa$?

I know Cohen's $\kappa$ for inter-rater reliability is often regarded as a conservative estimation--e.g., Strijbos & al. below, who say The higher the number of categories, the smaller the ...
1
vote
1answer
741 views

Kendall's coefficient of concordance (W) for ratings with a lot of ties

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As ...
1
vote
0answers
78 views

Looking for agreement amongst respondents

I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of ...
3
votes
0answers
254 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
0
votes
0answers
170 views

interrater reliability for multiple variables

i and another coder want to assess our agreement. our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
2
votes
1answer
32 views

Macro and Micro Inter-Coder/Rater Agreement

I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in ...
5
votes
1answer
111 views

Calculating inter rater reliability where raters and ratees only partially overlap

I'm new to inter rater reliability calculations. I have 5 developers and 6 raters. The raters ranked the first 3 of these developers based on some criteria like the following example: For ...
2
votes
1answer
121 views

How to assess relationship between responses to multiple likert items between two raters when there are multiple rater pairs?

I have two data sets in which the same Likert scale questions were answered by two different responders. I want to calculate which level they are correlated. My data set is like ...
4
votes
1answer
164 views

How to evaluate differences of multiple Likert scale ratings of multiple items within groups?

I have data where different reviewers rated how important multiple topics are on a scale of 1 to 5 (Likert scale). Each of the topics is assigned to a group. I want to see if there any differences in ...
1
vote
0answers
107 views

What is the proper test for inter-rater reliability (three raters, ordinal data)

I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal ...
1
vote
0answers
350 views

Which correlation analysis is appropriate for binary and ordinal variables?

Violin players were asked to rate 5 violins from least to most rich on a [0 1], 0.05-increment scale, ...
7
votes
2answers
339 views

Handling poor inter-rater reliability while minimizing the loss of data

Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
2
votes
0answers
82 views

Assessing the ability of two quantitative methods measuring the same thing

Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
1
vote
1answer
51 views

Test for treatment change

I have a study where the goal is to see if a new diagnostic tool provides any addition to the old procedure. A set of patients have been diagnosed and classified into 7 categories, each corresponding ...
2
votes
1answer
222 views

Basic Inter-rater reliability and type of data

I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability. Research question: Functional Movement Screening: How reliable are we in our ...
1
vote
2answers
443 views

Determining the number of raters for inter-rater reliability

I intend to ask a number of experts to match five out of a possible ten team roles (recorder, chairman and so on) to five distinct stages of a project. I cannot seem to find any source for ...
1
vote
0answers
127 views

Testing marginal homogeniety with multivariate outcomes and matched pairs

Suppose I have 20 people matched in 10 pairs. Suppose two doctors are assigned at random to diagnose one person in each pair, so in each pair each person is diagnosed by a different doctor. ...
3
votes
2answers
449 views

Inconsistency of findings between Kappa and ICC for IRR study

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
2
votes
2answers
79 views

Difference in proportions but one of them is 0

What is the best way to test for difference of proportions when of them is 0% (or 100%) ? edit: I have two raters, who give a score of either 1 or 0. One of them gave a 0 in every single case. Now I ...
2
votes
0answers
118 views

Testing graded response model thresholds for significance?

I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
1
vote
0answers
264 views

How should I interpret the Kappa Fleiss test?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
2
votes
0answers
99 views

Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?

For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
1
vote
1answer
180 views

Need normal distribution for kappa Fleiss and ICC (2, 1)?

I'm analysing a study of 4 raters and 34 subjects. I'm looking at the inter rater reliability. For the ordinal results, I did a weighted kappa of Fleiss and for the other continuous results I ...
2
votes
0answers
58 views

Relative scoring system without much effect from outliers

I am trying to assign scores to phone's specifications to calculate a final rating score for each phone. However, when I was scoring based on megapixels. I found that one of the phones 'A' has a 41MP ...
2
votes
0answers
72 views

What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?

I found a question that is related here, but it doesn't really goes on what I want to know. I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...
7
votes
1answer
281 views

What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
20
votes
2answers
567 views

Is Joel Spolsky's “Hunting of the Snark” post valid statistical content analysis?

If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He ...