Inter-rater reliability refers to the agreement (often measured by correlation) between two raters when rating a set of items. It is one part of properly developing a measure.

learn more… | top users | synonyms

0
votes
0answers
12 views

Calculating interrater agreement for multiple choice questions

I'm trying to figure out how to calculate interrater agreement so I can aggregate raters' responses. I'm asking 5 raters to select topics (out of 6 topics: A, B, C, D, E or F) that describe texts ...
1
vote
0answers
5 views

Inter-raters reliability for sentiment analysis lexicon

I have a list of words (more than 2000) which were rated by 4 raters for sentiments like anger (scale from 1 to 5). I run an alpha test on SPSS which gives me a coefficient of >8 which shows a good ...
0
votes
0answers
23 views

Interpretation of graph of pairwise concordance correlations

I have graphed pairwise concordance correlation coefficients (CCC) where a value of -1 should be complete disagreement, 1 perfect agreement, and values around 0.4 "good to fair". My results looks like ...
1
vote
0answers
27 views

Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
1
vote
0answers
12 views

Standard Error of Measurement (SEM) in Inter-rater reliability

I am currently writing my thesis about inter-rater reliability of a diagnostic tool between raters. I want to use the standard error of measurement with the formula: SEM of rater 1 and rater 2 = SD * ...
1
vote
1answer
78 views

Power analysis for inter-rater reliability study (Kappa) with multiple raters

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of ...
0
votes
0answers
22 views

Testing for intra-subject agreement across categories

I am looking at whether there is agreement between a subject's standardized weight (light-for-age or heavy-for-age) in early life and adulthood. I have coded light-for-age as '1' and heavy-for-age as ...
6
votes
1answer
49 views

Visualizing interrater disagreement

I am trying to visualize the process of two raters who have each rated the same dataset. Each row (line in the plot) represent a heart beat (or an error) on an ecg. Every disagreement was discussed ...
1
vote
1answer
33 views

Inter-rater reliability or another method?

I've asked about the mechanics of Inter-rater reliability before, but I guess I should have asked this question instead: If I have a bunch of raters and individuals (who are being judged on ...
0
votes
0answers
7 views

How many raters do I need to determine reliability of agreement on stimulus classification?

I have a stimulus set comprised of 147 videos, each with one of seven labels (hidden from raters). I want to determine how reliable label assignment is for each stimulus -- that is, I want to be able ...
1
vote
0answers
19 views

How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings ...
0
votes
1answer
64 views

Calculation for inter-rater reliability where raters don't overlap and different number per candidate?

I want to calculate the degree to which the gymnastics judges agree on balance beam scores, i.e., "inter-rater reliability". However, not all judges judge the same candidates, and the number of judges ...
0
votes
0answers
20 views

Kappas calculated using different R packages (epibasix vs. irr); weighted & unweighted kappa

I'm having some issues calculating inter-rater agreement in R. I need to calculate Cohen's kappas (quadratic weighted and unweighted). I've used the epibasix package to calculate unweighted kappas and ...
0
votes
0answers
16 views

Inter-rater reliability on multiple-choice tests

I have a task where coders are asked to assign categories to items by answering multiple choice questions. For each question, a coder is presented with an item and 5 categories (the total number of ...
2
votes
2answers
29 views

rubrics and analyzing essay scores

I have essay scores from one experiment. I have two raters and each rater separately graded the essays by using rubrics. The rubrics consist of 5 criteria, rating scales (1-5) and ...
0
votes
0answers
5 views

Test of Independence of raters in an assesment exercise

I am looking into data regarding a (quite subjective) essay scoring system we are dealling with. Let GPA be the overall gpa of the student and GPA_O, GPA_I and GPA_E be independent scores obtained ...
0
votes
0answers
10 views

calculate inter-rater agreement when you ask two raters to rank a set of things in preferred order of preference

I am new to statistics and currently trying to calculate the inter rater agreement between two entities based on their rankings of a set of things. For example if i give person 1 a set of 5 things and ...
0
votes
0answers
18 views

How do I calculate the inter-rater agreement for sequence alignments?

I am having this task of aligning one sequence of sentences to another sequence of sentences. Let's say we have sequence 1 and sequence 2 as an example of the task: Sequence 1:  A B C D E  F ...
0
votes
0answers
12 views

Which of these forms of data should I choose to calibrate markers of differing generosity?

12 teachers are teaching 600 students. The 12 cohorts taught by these teachers range in size from 40 to 90 students, and we expect systematic differences between cohorts, as graduate students were ...
0
votes
1answer
28 views

Inter-rater reliability

In my study a dog's owner and dog's walker each filled out a personality assessment for a target dog. I have 60 dogs in total. There were multiple questions that scored the dog on 5 dimensions. I'm ...
0
votes
0answers
16 views

Inter-rater reliability for two raters, multiple binary codes

I have two raters who utilized a binary codebook with codes (A, B, C) to code a series of responses. The codes are not independent, and a response could have more than one code (or no codes at all). ...
3
votes
0answers
50 views

Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
3
votes
0answers
25 views

Inter-rater agreement of a gold standard dataset - a ceiling for reliable evaluation of algorithms?

In my field, a dated gold standard dataset is used to track progress in algorithm development. Now when the state-of-the-art algorithms obtain higher correlation than is the inter-rater agreement of ...
0
votes
1answer
25 views

Rater's agreement for each item

I would like to calculate the agreement between 2 or more raters on all the judgments from a set, but also for each judgment. For example: ...
0
votes
1answer
17 views

Inter-rater reliability: two raters rate dataset A and two raters rate dataset B how to combine it into one number?

Me and one other person (Alice) counted the amount of boys and girls playing on a school playground (consent of parents and school were given of it being videotaped). Two other persons (Bob and ...
0
votes
1answer
12 views

How to measure correlations/congruence between parent and teacher ratings of children's personality

I was wondering if anyone could help. I am investigating children's personality, and I have collected both parents' and teachers' ratings for children on a 24-item personality scale of the big-five ...
0
votes
0answers
19 views

Fleiss Kappa: am I using it correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
0
votes
0answers
12 views

Raters' agreement in only one item judged

I've been looking for how can I measure the agreement between raters if there is only one item. I have already tried the measures of Fleiss' Kappa, Kendall's W and Krippendorff’s alpha. However, these ...
0
votes
0answers
26 views

What method(s) exist(s) for the classification and reliability (Cohen's kappa) analysis of partial data?

I'm trying to set up a study in which I have many (about 700) images to classify. I need to measure expert judgements on this data set. Based on literature on the same subject, I am setting up a study ...
1
vote
1answer
107 views

Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
0
votes
0answers
18 views

How do you show adequate inter-rater agreement?

I have data from 230 individuals from different companies, who rated their perceptions of their individual companies' organizational climates (four different scales ranging from 7 questions to 4 ...
0
votes
0answers
16 views

How to calculate overall rater variability from inter- and intra-rater variability?

I need to have a rough estimate of the overall variability in a dataset of contours manually drawn by different people and in different sessions. I know from a previous study that inter-rater ...
0
votes
0answers
27 views

Cohen's kappa for individual variables

I have a dataset of data points. For each point, two different raters assigned exactly one of four possible categories. I'm familiar with how to compute Cohen's kappa for the entire dataset, however, ...
0
votes
0answers
23 views

Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
2
votes
0answers
40 views

Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
2
votes
0answers
52 views

How to test the inter-rater reliability of a nominal coding process that generates ordinal data?

A researcher studying hunger on a tropical island asks respondents, "What did you eat for lunch yesterday?" and the possible choices are "(4) Nothing; (3) a banana; (2) an unripe coconut; (1) a ripe ...
0
votes
0answers
14 views

Conceptual difference between ordinally reliable and nominally reliable inter-rater statistic, K-alpha?

Using Krippendorff's alpha allows one to calculate an inter-rater reliability statistic on nominal, ordinal, interval, ratio, and other types of data. Assume that Krippendorff's alpha is 0.6 when ...
0
votes
1answer
92 views

Testing two raters' time-point data for interrater reliability

My data consists of two raters interpreting one specific phenomenon to occur at different points in time (the observations are not paired, the raters actually identified different amounts of ...
7
votes
2answers
130 views

How can I use this data to calibrate markers with different levels of generosity in grading student papers?

12 teachers are teaching 600 students. The 12 cohorts taught by these teachers range in size from 40 to 90 students, and we expect systematic differences between the cohorts, as graduate students were ...
1
vote
0answers
20 views

Inter-rater reliability for $k$ raters with $0$ to $n$ of $n$ categories applying to each subject

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
1
vote
1answer
59 views

advice on a solution attempt: interrater reliability of time-point data

In my data, two coders annotated subjectively (but independently), when certain time-point phenomena (a specific turn in a movement pattern) occurred. The data for the first 14 seconds looks ...
1
vote
0answers
21 views

Kendall's W for random subjects (not raters)

I have 5 raters (A,B,C,D,E) who evaluated 60 images on 1 to 10 points scale. The images are split into two groups: half of them were easy, half of them vere difficult (1 is easy, 10 is difficult). I ...
1
vote
1answer
26 views

interrater reliability for proportional (not count) data

My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data ...
0
votes
0answers
32 views

Observer agreement in multilevel binary data

I have a set of participants indicating with drawn circles where they experience pain on a manikin of the human body. These drawings were then independently rated by two human raters: each of 29 body ...
1
vote
1answer
97 views

Inter-rater reliability for unordered categorical data

I have some content analysis data that consist of unordered mutually exclusive categories as rated by two coders. What are useful approaches towards assessing inter-rater reliability for these data? ...
4
votes
1answer
455 views

Comparing inter-rater agreement between classes of raters

I'm interested in comparing ratings of the same objects that were done by raters from 2 different GROUPS/CLASSES (Experts, and Semi-Experts), so I can decide whether Semi-experts can replace experts ...
2
votes
0answers
80 views

Fleiss' Kappa alternative for inter-rater reliability with multiple answers

I have a dataset where researchers assign a binary yes/no answers to cells in a matrix. For each row more than one, but at least one, 'x' is allowed. For example, researchers 1 may code: ...
1
vote
0answers
39 views

A measure of correspondence between ranked ordinal data

I would like to find an appropriate way to measure the similarity between two sets of data with the following characteristics: Both sets contain 10 categorical observations. The categories ...
13
votes
5answers
489 views

How can I best deal with the effects of markers with differing levels of generosity in grading student papers?

Around 600 students have a score on an extensive piece of assessment, which can be assumed to have good reliability/validity. The assessment is scored out of 100, and it's a multiple-choice test ...
0
votes
1answer
49 views

What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...