Inter-rater reliability refers to the agreement (often measured by correlation) between two raters when rating a set of items. It is one part of properly developing a measure.

learn more… | top users | synonyms

0
votes
0answers
15 views

Is there a way to calculate inter-rater reliability (e.g. AC1 or Krippendorf's) for individual items when using nominal (categorical) data. [duplicate]

I am trying to generate a chance corrected measure of inter-rater reliability to assess items in a measure I have developed. I have agreement data from 25 expert judges. The measure assesses emotion ...
1
vote
1answer
14 views

R: Visualizing accuracy (inter-rater) on continuous data

Please excuse my english, I'm not a native speaker. So I'm running a test-retest scenario to determine accuracy of my measurements of continuous data (angles [deg] and length [mm]). I've calculated "...
0
votes
1answer
12 views

Inter-rater reliability simultaneously on multiple binary data

I have drug prescription records on 5 different drugs (A,B,C,D,E) on 2 different time points over a bunch of patients. 1/0 stands for use or not-use the specific drug, respectively. My questions are,...
1
vote
1answer
22 views

Intuition about f1 score

Let's say I have a data set where half of the data points are labelled as positive and half are labelled as negative. My task is to create a classifier which recognises when a sample from the dataset ...
1
vote
1answer
30 views

Weighted Fleiss' Kappa for Interval Data

I am looking for a variant of Fleiss' Kappa to deal with interval data, rather than strictly nominal/ordinal data. The context that I intend to use it in is as follows: There are several (5-8) ...
1
vote
1answer
21 views

After calculating inter-rater reliability, which data (e.g. consensus data?) should be used for further analyses?

My study used 2 raters in a fully crossed design to code 1) a categorical variable and 2) a continuous variable (i.e counting the number of times something happened). My question is, after ...
0
votes
0answers
9 views

Intraclass correlation, multiple items per person

I am evaluating inter-rater reliability for acoustic measurements (formant frequencies) for vowel sounds. Data description: Subjects are recorded speaking 28 words, 7 words (5 different words, 2 ...
0
votes
0answers
35 views

Calculate Cronbach's alpha for three observers

I have a dataset which contains 100 observations, and I asked three expert to describe each observation by one of the letters {A,B,C}. The following three tables shows a sample of the dataset with the ...
0
votes
0answers
10 views

Compute interrater reliability having different subsets of students rated by different subsets of raters

I would like to compute inter-rater reliability for a set of continuous items, having different subsets students assessed by different subsets of raters (please, see the picture). I would like to ...
1
vote
3answers
61 views

Correlation of two categorical variables

I have a list of N people, with two equivalent test performed (A and B), which can have as outcome 0 or 1 (cancer positive or negative). Test A is the accepted gold standard. Is there any way to find ...
1
vote
1answer
12 views

Randomization scheme for rating images and assessing inter-rater reliability

Apologies for the basic question, but I'm new to the area and I'm working on a research project on rating images, and assessing the reliability of these ratings, with the following structure: There ...
1
vote
2answers
47 views

Inter-subject agreement re timings of (unequal number of) events

I am trying to figure out what would be the best statistic to use to quantify the amount of agreement that exists between subjects who were asked to press a button whenever they felt a certain emotion ...
0
votes
1answer
49 views

Intraclass correlation coefficient: zero and negative

I am trying to calculate reliability between two raters for continuous data. The data is frequency of negative life events for each participant. My data looks like this: Dataset 1 rater 1: 0 0 0 0 0 ...
0
votes
1answer
48 views

How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
0
votes
1answer
111 views

Inter-rater reliability measure with multiple categories per item [duplicate]

I have multiple raters providing categorical ratings. What makes this slightly different from other posted questions is that each rater can provide three, unique categorical ratings. Each rater had ...
0
votes
0answers
29 views

Comparing two sets of limits of agreement (continuous outcome) - R

I am planning an analysis where I will have children and parents rating the child's pain (numerical score). For each child, I will have a child pain score and a parent pain score. The dyads will ...
0
votes
1answer
28 views

Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
0
votes
1answer
40 views

Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
3
votes
1answer
82 views

Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Stats newbie here, please bear with me. I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/...
0
votes
1answer
47 views

Calculating interrater agreement for multiple choice questions [duplicate]

I'm trying to figure out how to calculate interrater agreement so I can aggregate raters' responses. I'm asking 5 raters to select topics (out of 6 topics: A, B, C, D, E or F) that describe texts ...
1
vote
0answers
9 views

Inter-raters reliability for sentiment analysis lexicon

I have a list of words (more than 2000) which were rated by 4 raters for sentiments like anger (scale from 1 to 5). I run an alpha test on SPSS which gives me a coefficient of >8 which shows a good ...
0
votes
0answers
40 views

Interpretation of graph of pairwise concordance correlations

I have graphed pairwise concordance correlation coefficients (CCC) where a value of -1 should be complete disagreement, 1 perfect agreement, and values around 0.4 "good to fair". My results looks like ...
0
votes
0answers
47 views

Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
1
vote
1answer
81 views

Standard Error of Measurement (SEM) in Inter-rater reliability

I am currently writing my thesis about inter-rater reliability of a diagnostic tool between raters. I want to use the standard error of measurement with the formula: SEM of rater 1 and rater 2 = SD * ...
1
vote
1answer
401 views

Power analysis for inter-rater reliability study (Kappa) with multiple raters

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of ...
0
votes
1answer
31 views

Testing for intra-subject agreement across categories

I am looking at whether there is agreement between a subject's standardized weight (light-for-age or heavy-for-age) in early life and adulthood. I have coded light-for-age as '1' and heavy-for-age as '...
6
votes
1answer
54 views

Visualizing interrater disagreement

I am trying to visualize the process of two raters who have each rated the same dataset. Each row (line in the plot) represent a heart beat (or an error) on an ecg. Every disagreement was discussed ...
1
vote
1answer
44 views

Inter-rater reliability or another method?

I've asked about the mechanics of Inter-rater reliability before, but I guess I should have asked this question instead: If I have a bunch of raters and individuals (who are being judged on ...
0
votes
0answers
12 views

How many raters do I need to determine reliability of agreement on stimulus classification?

I have a stimulus set comprised of 147 videos, each with one of seven labels (hidden from raters). I want to determine how reliable label assignment is for each stimulus -- that is, I want to be able ...
1
vote
1answer
63 views

How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings (...
0
votes
1answer
76 views

Calculation for inter-rater reliability where raters don't overlap and different number per candidate?

I want to calculate the degree to which the gymnastics judges agree on balance beam scores, i.e., "inter-rater reliability". However, not all judges judge the same candidates, and the number of judges ...
0
votes
1answer
53 views

How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR. The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254. Is using Fleiss ...
0
votes
0answers
46 views

Kappas calculated using different R packages (epibasix vs. irr); weighted & unweighted kappa

I'm having some issues calculating inter-rater agreement in R. I need to calculate Cohen's kappas (quadratic weighted and unweighted). I've used the epibasix package to calculate unweighted kappas and ...
0
votes
0answers
37 views

Inter-rater reliability on multiple-choice tests

I have a task where coders are asked to assign categories to items by answering multiple choice questions. For each question, a coder is presented with an item and 5 categories (the total number of ...
2
votes
2answers
35 views

rubrics and analyzing essay scores

I have essay scores from one experiment. I have two raters and each rater separately graded the essays by using rubrics. The rubrics consist of 5 criteria, rating scales (1-5) and "descriptions"(...
0
votes
0answers
6 views

Test of Independence of raters in an assesment exercise

I am looking into data regarding a (quite subjective) essay scoring system we are dealling with. Let GPA be the overall gpa of the student and GPA_O, GPA_I and GPA_E be independent scores obtained ...
0
votes
1answer
13 views

How can I calculate inter-rater agreement for two raters assigning ranks?

I am new to statistics and currently trying to calculate the inter rater agreement between two entities based on their rankings of a set of things. For example if i give person 1 a set of 5 things and ...
0
votes
0answers
20 views

How do I calculate the inter-rater agreement for sequence alignments?

I am having this task of aligning one sequence of sentences to another sequence of sentences. Let's say we have sequence 1 and sequence 2 as an example of the task: Sequence 1:  A B C D E  F ...
0
votes
0answers
16 views

Which of these forms of data should I choose to calibrate markers of differing generosity?

12 teachers are teaching 600 students. The 12 cohorts taught by these teachers range in size from 40 to 90 students, and we expect systematic differences between cohorts, as graduate students were ...
0
votes
2answers
42 views

Which ICC should I use for inter-rater reliability?

In my study a dog's owner and dog's walker each filled out a personality assessment for a target dog. I have 60 dogs in total. There were multiple questions that scored the dog on 5 dimensions. I'm ...
1
vote
0answers
36 views

Inter-rater reliability for two raters, multiple binary codes

I have two raters who utilized a binary codebook with codes (A, B, C) to code a series of responses. The codes are not independent, and a response could have more than one code (or no codes at all). ...
3
votes
0answers
72 views

Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
3
votes
0answers
39 views

Inter-rater agreement of a gold standard dataset - a ceiling for reliable evaluation of algorithms?

In my field, a dated gold standard dataset is used to track progress in algorithm development. Now when the state-of-the-art algorithms obtain higher correlation than is the inter-rater agreement of ...
0
votes
2answers
48 views

Rater's agreement for each item

I would like to calculate the agreement between 2 or more raters on all the judgments from a set, but also for each judgment. For example: ...
0
votes
1answer
22 views

Inter-rater reliability: two raters rate dataset A and two raters rate dataset B how to combine it into one number?

Me and one other person (Alice) counted the amount of boys and girls playing on a school playground (consent of parents and school were given of it being videotaped). Two other persons (Bob and Claire)...
0
votes
2answers
22 views

How to measure correlations/congruence between parent and teacher ratings of children's personality

I was wondering if anyone could help. I am investigating children's personality, and I have collected both parents' and teachers' ratings for children on a 24-item personality scale of the big-five (...
0
votes
1answer
33 views

Am I using Fleiss' kappa coefficient correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
0
votes
0answers
16 views

Raters' agreement in only one item judged [duplicate]

I've been looking for how can I measure the agreement between raters if there is only one item. I have already tried the measures of Fleiss' Kappa, Kendall's W and Krippendorff’s alpha. However, these ...
1
vote
1answer
190 views

Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
0
votes
1answer
41 views

Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...