Inter-rater reliability refers to the agreement (often measured by correlation) between two raters when rating a set of items. It is one part of properly developing a measure.

learn more… | top users | synonyms

0
votes
0answers
15 views

type of ICC coefficient to use in my study for inter und intrarater-reliability

I am new to the subject and not sure, which ICC I have to use in order to analyse inter/intrarater-reliability of my measurements. A physician examined 649 healthy individuals with an ...
0
votes
0answers
12 views

Comparison test of reapeated Cohen's kappa

Suppose two raters judged a set of stimuli. Then Cohen's kappa was calculated as measure of inter-rater agreement ($\kappa_{pre-training}$). Suppose now that after a training session, the same two ...
0
votes
0answers
19 views

Statistical test to measure correlation between inter-rater mistakes

I'm working on a project that involves collecting crowd-sourced labels for discussion forum posts. The task given to annotators was to look at a forum post and to decide which of six classes the post ...
0
votes
0answers
10 views

Inter-rater reliability

Two raters have scored essays based on five criteria. I would like to calculate inter-rater reliability (IRR). Should I calculate IRR on each criterion or just the total score of five criteria? ...
0
votes
0answers
50 views

Should I use both paired t-test and intraclass correlation coefficient to examine the interrater reliability of an instrument?

As title, I am assessing the interrater reliability of a questionnaire (actually a Barthel Index (BI)) performed by four assessors (A,B,C,D) on about 25 test subjects. I plan to examine the IRR from ...
0
votes
0answers
16 views

Interpreting the relInterIntra {irr} results in R

I am using this for a reliability analysis test for quantitative values between two raters. I got my results using the "relInterIntra {irr}" package in R, but the reported values gave me several ...
0
votes
1answer
15 views

What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
2
votes
0answers
34 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
0
votes
0answers
7 views

Which test do I run to check inter-rater agreement for multiple ordinal variables?

I'd like to check the inter-rater agreement between the six raters for the six ordinal variables scored for each of the 24 score sheets passed to each rater (6 ordinal variables per sheet, 24 sheets ...
1
vote
1answer
75 views

Do I run a one-tailed or two-tailed test when running Spearman's correlation between interval and ordinal data?

I have one interval variable on a 1-25 point interval scale, i.e. placement test scores where 25 is the highest competence score (data not normally distributed), and six different ordinal variables on ...
2
votes
0answers
17 views

Analysis of ordinal data with multiple assessments per unit… the right approach?

Suppose I have a dataset with the following information: N objects, each of which can be rated The number of ratings per object is variable (raters are unidentified), e.g., there are 30 objects each ...
1
vote
0answers
58 views

Can you test if one judge gives significantly different scores from other judges?

If I had a small number of observations (about 5) in a sample and the population standard deviation was unknown, what test could be done to see if the value of one of the observations was ...
1
vote
0answers
24 views

Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
0
votes
0answers
49 views

Measure of accuracy or measure of agreement for continuous data with true values

I have two continuous variables. Variable $X$ represents the true values (basically it is atmospheric pressure measured via a barometer). Variable $Y$ represents a way to approximate the atmospheric ...
0
votes
1answer
76 views

How will two raters score videos?

I have 260 teacher conferences to score using a rubric. How do two raters need to score these for rater agreement? Do we need three raters?
0
votes
0answers
46 views

Statistical formula for intra-rater reliability

I designed an assessment rubric for a class assignment and wish to calculate my own intra-rater reliability. My goal is to use the rubric to evaluate the same ordinal data at two different points in ...
0
votes
0answers
81 views

Inter-rater reliability: multiple raters and multiple replications

This is really a two part question. I have been asked to assess inter-rater agreement on a study in which 4 raters were asked to rate microarrays from patient samples. Ratings were based on a 5 point ...
1
vote
1answer
57 views

How to summarize measurement variability for repeated measures?

I have a data set where different measures are taken repeatedly (3) times for multiple samples, like so: A B C X ... Y ... So, for people X, Y, etc I have ...
0
votes
0answers
24 views

How can we measure between-experimenter reliability for experimental data?

Description of my situation: I will run an experiment in a few weeks with a total sample size of N = 70. Ideally I would prefer to have only one experimenter to collect all the data but I have to do ...
0
votes
0answers
77 views

Intraclass correlation (ICC) per item instead of across items

I am using the intraclass correlation coefficiant to determine inter-rater reliability for a set of pictures that were rated by several raters on a scale from 1-9. I am using the two-way mixed model ...
1
vote
0answers
123 views

Inter-rater agreement with multiple raters

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
2
votes
0answers
151 views

Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
3
votes
0answers
62 views

What is the analysis techniques for comparing two sets of ratings

Say, I have collected 50 comments. For each comment, I am planning to get 10 ratings from non-expert raters for the following criteria: "interesting vs. not interesting". Based on the plurality of ...
2
votes
1answer
1k views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
3
votes
2answers
6k views

Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS. Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important). All ...
1
vote
0answers
64 views

Is Fleiss's $\kappa$ as conservative as Cohen's $\kappa$?

I know Cohen's $\kappa$ for inter-rater reliability is often regarded as a conservative estimation--e.g., Strijbos & al. below, who say The higher the number of categories, the smaller the ...
1
vote
1answer
809 views

Kendall's coefficient of concordance (W) for ratings with a lot of ties

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As ...
1
vote
0answers
80 views

Looking for agreement amongst respondents

I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of ...
3
votes
0answers
278 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
0
votes
0answers
197 views

interrater reliability for multiple variables

i and another coder want to assess our agreement. our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
2
votes
1answer
33 views

Macro and Micro Inter-Coder/Rater Agreement

I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in ...
5
votes
1answer
124 views

Calculating inter rater reliability where raters and ratees only partially overlap

I'm new to inter rater reliability calculations. I have 5 developers and 6 raters. The raters ranked the first 3 of these developers based on some criteria like the following example: For ...
2
votes
1answer
127 views

How to assess relationship between responses to multiple likert items between two raters when there are multiple rater pairs?

I have two data sets in which the same Likert scale questions were answered by two different responders. I want to calculate which level they are correlated. My data set is like ...
4
votes
1answer
171 views

How to evaluate differences of multiple Likert scale ratings of multiple items within groups?

I have data where different reviewers rated how important multiple topics are on a scale of 1 to 5 (Likert scale). Each of the topics is assigned to a group. I want to see if there any differences in ...
1
vote
0answers
111 views

What is the proper test for inter-rater reliability (three raters, ordinal data)

I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal ...
1
vote
0answers
409 views

Which correlation analysis is appropriate for binary and ordinal variables?

Violin players were asked to rate 5 violins from least to most rich on a [0 1], 0.05-increment scale, ...
7
votes
2answers
355 views

Handling poor inter-rater reliability while minimizing the loss of data

Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
2
votes
0answers
88 views

Assessing the ability of two quantitative methods measuring the same thing

Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
1
vote
1answer
55 views

Test for treatment change

I have a study where the goal is to see if a new diagnostic tool provides any addition to the old procedure. A set of patients have been diagnosed and classified into 7 categories, each corresponding ...
2
votes
1answer
245 views

Basic Inter-rater reliability and type of data

I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability. Research question: Functional Movement Screening: How reliable are we in our ...
1
vote
2answers
491 views

Determining the number of raters for inter-rater reliability

I intend to ask a number of experts to match five out of a possible ten team roles (recorder, chairman and so on) to five distinct stages of a project. I cannot seem to find any source for ...
1
vote
0answers
128 views

Testing marginal homogeniety with multivariate outcomes and matched pairs

Suppose I have 20 people matched in 10 pairs. Suppose two doctors are assigned at random to diagnose one person in each pair, so in each pair each person is diagnosed by a different doctor. ...
3
votes
2answers
486 views

Inconsistency of findings between Kappa and ICC for IRR study

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
2
votes
2answers
81 views

Difference in proportions but one of them is 0

What is the best way to test for difference of proportions when of them is 0% (or 100%) ? edit: I have two raters, who give a score of either 1 or 0. One of them gave a 0 in every single case. Now I ...
2
votes
0answers
123 views

Testing graded response model thresholds for significance?

I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
1
vote
0answers
271 views

How should I interpret the Kappa Fleiss test?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
2
votes
0answers
103 views

Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?

For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
1
vote
1answer
196 views

Need normal distribution for kappa Fleiss and ICC (2, 1)?

I'm analysing a study of 4 raters and 34 subjects. I'm looking at the inter rater reliability. For the ordinal results, I did a weighted kappa of Fleiss and for the other continuous results I ...
2
votes
0answers
61 views

Relative scoring system without much effect from outliers

I am trying to assign scores to phone's specifications to calculate a final rating score for each phone. However, when I was scoring based on megapixels. I found that one of the phones 'A' has a 41MP ...
2
votes
0answers
74 views

What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?

I found a question that is related here, but it doesn't really goes on what I want to know. I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...