Inter-rater reliability refers to the agreement (often measured by correlation) between two raters when rating a set of items. It is one part of properly developing a measure.

learn more… | top users | synonyms

0
votes
1answer
6 views

Inter-rater reliability: two raters rate dataset A and two raters rate dataset B how to combine it into one number?

Me and one other person (Alice) counted the amount of boys and girls playing on a school playground (consent of parents and school were given of it being videotaped). Two other persons (Bob and ...
0
votes
1answer
11 views

How to measure correlations/congruence between parent and teacher ratings of children's personality

I was wondering if anyone could help. I am investigating children's personality, and I have collected both parents' and teachers' ratings for children on a 24-item personality scale of the big-five ...
0
votes
0answers
8 views

Fleiss Kappa: am I using it correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
0
votes
0answers
6 views

Raters' agreement in only one item judged

I've been looking for how can I measure the agreement between raters if there is only one item. I have already tried the measures of Fleiss' Kappa, Kendall's W and Krippendorff’s alpha. However, these ...
0
votes
0answers
15 views

What method(s) exist(s) for the classification and reliability (Cohen's kappa) analysis of partial data?

I'm trying to set up a study in which I have many (about 700) images to classify. I need to measure expert judgements on this data set. Based on literature on the same subject, I am setting up a study ...
1
vote
0answers
27 views

Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
0
votes
0answers
11 views

How do you show adequate inter-rater agreement?

I have data from 230 individuals from different companies, who rated their perceptions of their individual companies' organizational climates (four different scales ranging from 7 questions to 4 ...
0
votes
0answers
11 views

How to calculate overall rater variability from inter- and intra-rater variability?

I need to have a rough estimate of the overall variability in a dataset of contours manually drawn by different people and in different sessions. I know from a previous study that inter-rater ...
0
votes
0answers
22 views

Cohen's kappa for individual variables

I have a dataset of data points. For each point, two different raters assigned exactly one of four possible categories. I'm familiar with how to compute Cohen's kappa for the entire dataset, however, ...
0
votes
0answers
15 views

Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
1
vote
0answers
25 views

Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
2
votes
0answers
16 views

How to test the inter-rater reliability of a nominal coding process that generates ordinal data?

A researcher studying hunger on a tropical island asks respondents, "What did you eat for lunch yesterday?" and the possible choices are "(4) Nothing; (3) a banana; (2) an unripe coconut; (1) a ripe ...
0
votes
0answers
9 views

Conceptual difference between ordinally reliable and nominally reliable inter-rater statistic, K-alpha?

Using Krippendorff's alpha allows one to calculate an inter-rater reliability statistic on nominal, ordinal, interval, ratio, and other types of data. Assume that Krippendorff's alpha is 0.6 when ...
0
votes
1answer
61 views

Testing two raters' time-point data for interrater reliability

My data consists of two raters interpreting one specific phenomenon to occur at different points in time (the observations are not paired, the raters actually identified different amounts of ...
7
votes
2answers
118 views

How can I use this data to calibrate markers with different levels of generosity in grading student papers?

12 teachers are teaching 600 students. The 12 cohorts taught by these teachers range in size from 40 to 90 students, and we expect systematic differences between the cohorts, as graduate students were ...
1
vote
0answers
15 views

Inter-rater reliability for $k$ raters with $0$ to $n$ of $n$ categories applying to each subject

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
1
vote
1answer
45 views

advice on a solution attempt: interrater reliability of time-point data

In my data, two coders annotated subjectively (but independently), when certain time-point phenomena (a specific turn in a movement pattern) occurred. The data for the first 14 seconds looks ...
1
vote
0answers
14 views

Kendall's W for random subjects (not raters)

I have 5 raters (A,B,C,D,E) who evaluated 60 images on 1 to 10 points scale. The images are split into two groups: half of them were easy, half of them vere difficult (1 is easy, 10 is difficult). I ...
1
vote
1answer
16 views

interrater reliability for proportional (not count) data

My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data ...
0
votes
0answers
21 views

Observer agreement in multilevel binary data

I have a set of participants indicating with drawn circles where they experience pain on a manikin of the human body. These drawings were then independently rated by two human raters: each of 29 body ...
1
vote
1answer
72 views

Inter-rater reliability for unordered categorical data

I have some content analysis data that consist of unordered mutually exclusive categories as rated by two coders. What are useful approaches towards assessing inter-rater reliability for these data? ...
3
votes
1answer
258 views

Comparing inter-rater agreement between classes of raters

I'm interested in comparing ratings of the same objects that were done by raters from 2 different GROUPS/CLASSES (Experts, and Semi-Experts), so I can decide whether Semi-experts can replace experts ...
0
votes
0answers
52 views

Fleiss' Kappa alternative for inter-rater reliability with multiple answers

I have a dataset where researchers assign a binary yes/no answers to cells in a matrix. For each row more than one, but at least one, 'x' is allowed. For example, researchers 1 may code: ...
1
vote
0answers
32 views

A measure of correspondence between ranked ordinal data

I would like to find an appropriate way to measure the similarity between two sets of data with the following characteristics: Both sets contain 10 categorical observations. The categories ...
13
votes
5answers
480 views

How can I best deal with the effects of markers with differing levels of generosity in grading student papers?

Around 600 students have a score on an extensive piece of assessment, which can be assumed to have good reliability/validity. The assessment is scored out of 100, and it's a multiple-choice test ...
0
votes
0answers
48 views

type of ICC coefficient to use in my study for inter und intrarater-reliability

I am new to the subject and not sure, which ICC I have to use in order to analyse inter/intrarater-reliability of my measurements. A physician examined 649 healthy individuals with an ...
0
votes
0answers
94 views

Comparison test of reapeated Cohen's kappa

Suppose two raters judged a set of stimuli. Then Cohen's kappa was calculated as measure of inter-rater agreement ($\kappa_{pre-training}$). Suppose now that after a training session, the same two ...
0
votes
0answers
33 views

Statistical test to measure correlation between inter-rater mistakes

I'm working on a project that involves collecting crowd-sourced labels for discussion forum posts. The task given to annotators was to look at a forum post and to decide which of six classes the post ...
0
votes
0answers
25 views

Inter-rater reliability

Two raters have scored essays based on five criteria. I would like to calculate inter-rater reliability (IRR). Should I calculate IRR on each criterion or just the total score of five criteria? ...
0
votes
0answers
214 views

Should I use both paired t-test and intraclass correlation coefficient to examine the interrater reliability of an instrument?

As title, I am assessing the interrater reliability of a questionnaire (actually a Barthel Index (BI)) performed by four assessors (A,B,C,D) on about 25 test subjects. I plan to examine the IRR from ...
0
votes
0answers
24 views

Interpreting the relInterIntra {irr} results in R

I am using this for a reliability analysis test for quantitative values between two raters. I got my results using the "relInterIntra {irr}" package in R, but the reported values gave me several ...
0
votes
1answer
38 views

What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
2
votes
0answers
72 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
1
vote
1answer
938 views

Do I run a one-tailed or two-tailed test when running Spearman's correlation between interval and ordinal data?

I have one interval variable on a 1-25 point interval scale, i.e. placement test scores where 25 is the highest competence score (data not normally distributed), and six different ordinal variables on ...
2
votes
0answers
26 views

Analysis of ordinal data with multiple assessments per unit… the right approach?

Suppose I have a dataset with the following information: N objects, each of which can be rated The number of ratings per object is variable (raters are unidentified), e.g., there are 30 objects each ...
1
vote
0answers
91 views

Can you test if one judge gives significantly different scores from other judges?

If I had a small number of observations (about 5) in a sample and the population standard deviation was unknown, what test could be done to see if the value of one of the observations was ...
1
vote
0answers
37 views

Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
0
votes
0answers
139 views

Measure of accuracy or measure of agreement for continuous data with true values

I have two continuous variables. Variable $X$ represents the true values (basically it is atmospheric pressure measured via a barometer). Variable $Y$ represents a way to approximate the atmospheric ...
0
votes
2answers
113 views

How will two raters score videos?

I have 260 teacher conferences to score using a rubric. How do two raters need to score these for rater agreement? Do we need three raters?
1
vote
1answer
76 views

How to summarize measurement variability for repeated measures?

I have a data set where different measures are taken repeatedly (3) times for multiple samples, like so: A B C X ... Y ... So, for people X, Y, etc I have ...
2
votes
0answers
158 views

Inter-rater agreement with multiple raters

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
2
votes
0answers
219 views

Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
3
votes
0answers
85 views

What is the analysis techniques for comparing two sets of ratings

Say, I have collected 50 comments. For each comment, I am planning to get 10 ratings from non-expert raters for the following criteria: "interesting vs. not interesting". Based on the plurality of ...
2
votes
1answer
3k views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
3
votes
2answers
11k views

Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS. Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important). All ...
1
vote
0answers
80 views

Is Fleiss's $\kappa$ as conservative as Cohen's $\kappa$?

I know Cohen's $\kappa$ for inter-rater reliability is often regarded as a conservative estimation--e.g., Strijbos & al. below, who say The higher the number of categories, the smaller the ...
1
vote
1answer
1k views

Kendall's coefficient of concordance (W) for ratings with a lot of ties

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As ...
1
vote
0answers
83 views

Looking for agreement amongst respondents

I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of ...
4
votes
0answers
410 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
0
votes
0answers
325 views

interrater reliability for multiple variables

i and another coder want to assess our agreement. our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...