Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
25
votes
2answers
26k views

Inter-rater reliability for ordinal or interval data

Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...
25
votes
2answers
821 views

Is Joel Spolsky's “Hunting of the Snark” post valid statistical content analysis?

If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He ...
13
votes
5answers
546 views

How can I best deal with the effects of markers with differing levels of generosity in grading student papers?

Around 600 students have a score on an extensive piece of assessment, which can be assumed to have good reliability/validity. The assessment is scored out of 100, and it's a multiple-choice test ...
13
votes
2answers
901 views

Interrater reliability for events in a time series with uncertainty about event time

I have multiple independent coders who are trying to identify events in a time series -- in this case, watching video of face-to-face conversation and looking for particular nonverbal behaviors (e.g., ...
9
votes
2answers
162 views

How can I use this data to calibrate markers with different levels of generosity in grading student papers?

12 teachers are teaching 600 students. The 12 cohorts taught by these teachers range in size from 40 to 90 students, and we expect systematic differences between the cohorts, as graduate students were ...
9
votes
1answer
2k views

Computing inter-rater reliability in R with variable number of ratings?

Wikipedia suggests that one way to look at inter-rater reliability is to use a random effects model to compute intraclass correlation. The example of intraclass correlation talks about looking at $$\...
8
votes
1answer
7k views

Quadratic weighted kappa

I have done a little Googling about quadratic weighted kappa, but I couldn't find a good explanation that make me understood that. Can somebody give some resource or brief explanation?
8
votes
4answers
6k views

Matthews correlation coefficient with multi-class

Matthews correlation coefficient ($\textrm{MCC}$) is a measurement to measure the quality of a binary classification ([Wikipedia][1]). $\textrm{MCC}$ formulation is given for binary classification ...
8
votes
2answers
6k views

What to do in case of low inter-rater reliability (ICC)?

Background: Eight doctors each rated the same 54 patients on a persuasiveness measure (1-7 Likert scale). The mean score on the persuasiveness measure will eventually be the outcome measure of my ...
8
votes
1answer
2k views

How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?

Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
8
votes
2answers
1k views

Inter-rater reliability with many non-overlapping raters

I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
7
votes
3answers
944 views

Inter-rater statistic for skewed rankings

I have several sets of 10 raters which I want to compare. Each rater can cast only Yes or No vote, however this decision is skewed and the Yes votes make only about 10% of all votes (and this is ...
7
votes
2answers
516 views

Handling poor inter-rater reliability while minimizing the loss of data

Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
7
votes
1answer
6k views

Creating and interpreting Bland-Altman plot

Yesterday I heard of the Bland-Altman plot for the first time. I have to compare two methods of measuring blood pressure, and I need to produce a Bland-Altman plot. I am not sure if I get everything ...
7
votes
3answers
3k views

Bland-Altman (Tukey Mean-Difference) plot for differing scales

I find that Bland-Altman plots for comparing two methods are extremely useful in assessing agreement. However, I'm curious if there is a similar method or transformation that can be used when the ...
7
votes
1answer
504 views

What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
6
votes
2answers
3k views

Repeatability and measurement error from and between observers

I have 3 observers that each take 2 measurements (length and weight) on 100 individuals; these procedures are repeated once (i.e., the same measurements are taken on the same 100 individuals by the ...
6
votes
1answer
4k views

What's the difference between concordance correlation and intraclass correlation?

I understand there are various formulae for calculating an intraclass correlation, and various formulae for calculating a concordance correlation. Is there something that all intraclass correlations ...
6
votes
1answer
7k views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
6
votes
2answers
143 views

How to index rater agreement: multiple raters identify strenghts/weaknesses from 30 traits?

Six raters independently rated 25 people by identifying strengths and weaknesses: choosing among 35 traits (15 positive traits and 20 negative traits). The rater instructions said to identify a ...
6
votes
1answer
189 views

Visualizing interrater disagreement

I am trying to visualize the process of two raters who have each rated the same dataset. Each row (line in the plot) represent a heart beat (or an error) on an ecg. Every disagreement was discussed ...
6
votes
0answers
377 views

Inter-rater agreement of a gold standard dataset - a ceiling for reliable evaluation of algorithms?

In my field, a dated gold standard dataset is used to track progress in algorithm development. Now when the state-of-the-art algorithms obtain higher correlation than is the inter-rater agreement of ...
5
votes
1answer
7k views

Intra- and inter-rater reliability on the same data

I've read some of the answers here (specifically the 'Inter-rater reliability for ordinal or interval data' one). I'm still somewhat perplexed though! I have data for 4 raters, who each rated every ...
5
votes
1answer
5k views

Adjusting kappa inter-rater agreement for prevalence

I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement. Can anyone advise on how ...
5
votes
1answer
11k views

Inter-rater reliability using Intra-class correlation with ratings for multiple objects on multiple properties

Initial Question: I am trying to calculate inter-rater reliability. Previous researchers in this area have used intraclass correlation. SPSS has options for two-way random, mixed random and one-way ...
5
votes
4answers
31k views

Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS. Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important). All ...
5
votes
4answers
955 views

Assessing rater bias where one rater has given one very high rating and the remainder very low ratings

What is a good statistical test to check if there is a bias in judging in a situation that there is one judge that gave extreme scores (high score for one of the contestant and very low scores on the ...
5
votes
1answer
275 views

Is there a specific name for this plot?

This plot represents the popularity of technologies in two "tools", vue and react. In left-top corner are specific technologies for vue but not for react, right-top technologies popular in both tools ...
5
votes
2answers
1k views

Compare a diagnostic test to gold standard

I have a sample of patients who underwent 2 diagnostic tests, one of which is the gold standard. I've been asked to test if the diagnosis from the experimental test is different than the one from the ...
5
votes
1answer
2k views

Get a 95% confidence interval for Cohen's Kappa in R

I've got a study with two radiologist's reading chest x-rays and want to calculate Cohen's kappa of their agreement. The kappa2 function in the "irr" package and <...
5
votes
1answer
865 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
5
votes
1answer
3k views

Comparing inter-rater agreement between classes of raters

I'm interested in comparing ratings of the same objects that were done by raters from 2 different GROUPS/CLASSES (Experts, and Semi-Experts), so I can decide whether Semi-experts can replace experts ...
5
votes
1answer
2k views

Inter-rater agreement for Likert scale

What is the best inter-rater agreement test for Likert scale type questions? As far as I see, Cronbach's $\alpha$ is for internal consistency, or it shows how good items are related to describe the ...
5
votes
1answer
277 views

Calculating inter rater reliability where raters and ratees only partially overlap

I'm new to inter rater reliability calculations. I have 5 developers and 6 raters. The raters ranked the first 3 of these developers based on some criteria like the following example: For example,...
5
votes
2answers
1k views

Combining ratings from multiple raters of different accuracy

I have a data set of J raters who each give ratings to I objects (on say a scale of 1 to 5). My goal would be to construct some kind of overall rating for each i based on all of the raters' scores. ...
4
votes
2answers
1k views

Inconsistency of findings between Kappa and ICC for IRR study

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
4
votes
2answers
5k views

Assessing and testing inter-rater agreement with kappa statistic on a set of binary and Likert items?

I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions. Should inter-...
4
votes
1answer
3k views

Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
4
votes
1answer
595 views

Testing for similarity on count data

Is there a statistical test tat can test for similarity between two data sets on a count data? The only test I can think of doing on a count data is chi squared test and fisher exact test and both ...
4
votes
2answers
54 views

Quantifying agreement between datasets from different methods of measurement

I have two datasets, A and B. Both ultimately come from the same source, both have the same data attributes, but the datasets use different measurement methods to derive the attributes. We want to ...
4
votes
1answer
2k views

Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Stats newbie here, please bear with me. I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/...
4
votes
1answer
69 views

How to determine which variable agrees best with a gold standard?

I have a dataset of 100 patients. Each patient has 3 lab tests: Lab Test 0: "Gold Standard" test which will serve as my reference standard Lab Test 1 Lab Test 2 I want to compare the differences ...
4
votes
1answer
652 views

Fisher's exact test vs kappa analysis

I was reading a paper where the authors assessed the association between two different diagnostics tests intended to diagnose the same disease and they performed the analysis with Fisher's exact test. ...
4
votes
1answer
734 views

How should I interpret Fleiss' kappa when it equals NaN?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
4
votes
2answers
361 views

How to “prove” that new measurement tool & process gives same result as old?

I am new to the area of statistics and I am hoping you can suggest methods I may use. Sorry if this is long but I might as well be as clear as possible on my first post :) What I am worried most is ...
4
votes
1answer
2k views

Kendall's coefficient of concordance (W) for ratings with a lot of ties

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As ...
4
votes
1answer
241 views

How to evaluate differences of multiple Likert scale ratings of multiple items within groups?

I have data where different reviewers rated how important multiple topics are on a scale of 1 to 5 (Likert scale). Each of the topics is assigned to a group. I want to see if there any differences in ...
4
votes
1answer
1k views

Interrater reliability in SPSS

I am trying to calculate interrater reliability in SPSS for both pre and post test of the same measure that is administered as part of a prison intake program. The measure has 20 items and then a ...
4
votes
1answer
218 views

Ordered category agreement

I have data on two processes, where the process assigns elements into ordered bins. I am interested in testing for agreement between the processes. What is the best way to do this (R code)? Here is ...
4
votes
2answers
88 views

Evaluating TAs grading

Let me introduce the context: I need to evaluate how my TAs tend to grade students. Ideally I would like to pinpoint whether they show some sort of internal consistency (e.g., a TA usually grades ...

1 2 3 4 5 6