# Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

397 questions
Filter by
Sorted by
Tagged with
41 views

### assessing which random sample agrees more with a preferred ranking

I am not a mathematician or a statistician. But, I think the question I have is related to statistics. I will start with a made up example. If I can grade apples into,say four grades from 1 to 4, one ...
• 23
16 views

### Can there be a single limit of agreement value in Bland-Altman analysis?

While trying to learn Bland-Altman analysis, I came across a situation that confused me a bit. I can understand upper (mean + 2 SD) and lower limits (mean – 2 SD), but a single limit of agreement (LOA)...
• 51
1 vote
29 views

### How to determine agreement between clustering methods?

Let's say you want to compare the outcome of KMeans and KMedoids. How to determine if cluster 1 from KMeans can be compared with cluster 1 with KMedoids. Or, in other words, let's say KMeans labels ...
1 vote
36 views

### Can I compare Cohen's and Fleiss' kappa values?

I am running an experiment where a number of human specialists perform a task by hand, and then a computer algorithm performs the same task. I want to determine how similar the algorithm's results are ...
• 111
16 views

### Best statistical test and power calculation for check of interpretation

What is the best statistical test for the following situation. A radiologist interprets chest X-rays (CXRs) of a population. His read of the CXRs is either Yes there is disease or No there is not ...
• 101
29 views

### Inter-rater reliability for 2 coders of 30 responses each for 100 different items

I have a data set with the structure outlined below which was coded by two independent raters and I want to determine inter-rater reliability. The dataset contains responses to 100 different items by ...
• 1
21 views

### Calculate interrater reliability for multiple trials and raters in R

I have data for a large number of Trials (only three shown here) and ratings by subjects A, B...
1 vote
45 views

### Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
• 13
16 views

### When is it more appropriate to use linear or quadratic weights to measure inter-rater reliability (Fleiss's kappa or Gwet's AC1?) [duplicate]

I am analysing the results of a linguistic acceptability judgment study where 36 speakers had to rate a finite number of sentences on a 7-point Likert scale (so an ordinal scale, although I am not ...
43 views

### Using and Reporting ICC for Aggregated Data About Groups of People

I've designed a survey that consists of multiple questionnaires with Likert-scale questions, each of which targets a specific variable in my research model. The participants of this survey are ...
53 views

### Inter annotator agreement when the number of annotators is not consistent across the samples [duplicate]

I have a sample of around 100 that has been annotated as either 0, 0,5, or 1 by 5 annotators. For some reason, I found that some of the annotations need to be deleted. Thus, for some samples, we have ...
• 135
25 views

### appropriate ICC

My apologies this is a rookie question, I am aware of what ICC(Intraclass correlation) is, and that it measures how strongly subjects in the same group resemble each other. However I am not sure how ...
1 vote
26 views

### Quantifying agreement of ordinally ranked sequences

I've been using a very particular set of closely related metrics to quantify how well two ranked sequences agree with one another. I'd like to know if this way of thinking is well-known and if any of ...
18 views

### interrater reliability with 12 pairs parents rating their own children in categorical variables

Please accept my apologies if the answer to this question already exists. This is the first time for me to do a study investigating the interrater reliability of a questionnaire. There are 12 pairs of ...
• 1
42 views

### Inter Annotator Agreement for Question Answering

I would like to run an Inter Annotator Agreement (IAA) test for Question Answering. I've tried to look for a method to do it, but I wasn't able to get exactly what I need. I've read that there are ...
7 views

### Survey analysis and data handling

I have survey responses from 67 raters. They responded to 7 clinical scenarios. Each scenario had a set of possible responses which are not numeric and not ordinal. A response can be one of 10 ...
27 views

### What is the correct inter-rater reliability measure for 2 raters annotating video frames with one of 3 categories?

In my problem, two raters are given 30 videos. For the videos each second will be annotated with one of 3 categories (A making an error, B making an error, no error). For instance, a video could be ...
13 views

### Pearson Correlation for Inter and Intra-reliability coeffciient

I am reading this research paper entitled, "Inter-Rater Reliability and Intra-Rater Reliability of Assessing the 2-Minute Push-Up Test" since it is somewhat similar to what I am currently ...
25 views

### Measuring within-group similarity for ranked list responses

Individuals are completing a task in which they rank 15 items (1-15 according to importance). These individuals will later be working in groups of 4, so I am interested in measuring the similarity of ...
• 1
30 views

### Inter-annotator agreemnet on a dataset with no annotator id

I have a dataset of annotations like the following, ...
• 101
14 views

### Proper metric to compare continuous to discrete data? [duplicate]

I have two measurements of the same thing. The first is a subjective rating score, the second is a quantitative metric. The rating score is discrete (0-10), while the quantitative metric is any value ...
15 views

### How to compare scale ratings of multiple dimensions from two external raters and several participants who provide different numbers of responses?

I started with Krippendorff's Alpha but if I understand correctly, that assumes that ratings by participants are all by the same individual? Nearly 500 participants produced short pieces of writing ...
14 views

### Test-retest reliability with varying number of measurements

I developed a new method to measure a certain quantity on a radiological image of a patient. I have available varying numbers of studies per patient. I applied the method to all images available for ...
9 views

### Kappa Agreement analysis for two related classifications

We conducted an experiment where participants compare a metric map and sketch map of the same area. In the sketch map, they are asked to classify generalized (yellow) and non-generalized (green)...
1 vote
47 views

### An a posteriori test to determine non difference across >3 groups given a skewed continuous variable of interest, as an alternative to a Friedman-test

Given a set of items of which each item is once measured using a computative model, each time under a different condition, what is the best way to quantify the magnitude of difference between ...
1 vote
90 views

### How to measure machine-human agreement in the light of human-human agreement

Suppose you have a set of subjects, and for each one a measurement of a certain property by a machine and measurements of the same property by several separate humans. Each measurement is a single ...
1 vote
60 views

### What is the correct nesting structure for calculating the Inter-rater reliability or intra class correlation (ICC) with multiple raters?

I need to figure out which inter-rater reliability for multiple raters I should calculate. I'm trying to figure out how I should nest or structure certain observations within my dataset in order to ...
79 views

### Checking the relation between two questions

I have set of answers obtained from different respondents to the following questions (with available answers): Q1 --> Do you think this practice is bad? A1 --> YES or NO. Q2 --> What is the ...
115 views

### Alternative to Intra-class correlation coefficient (ICC) for continuous data

I am working on a research project that involves a reliability study. In this case, 3 raters have assigned two sets of scores to 50 items: an ordinal score on a scale from 1 - 3, and a continuous ...
26 views

### What measure of agreeement should I use for time series data?

I am trying to compare the agreement between two measurements. Each measurement tells me when a device is worn or not worn (binary) and is sampled at the same frequency (1Hz). The measurements were ...
1 vote
21 views

### Calculating the quality of text parsing script

I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research. The ...
• 11
1 vote
45 views

### Weighted Kappa with rating variables that have different set of categories

I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT Rater1 rated the the ...
• 467
21 views

### Two raters’ inter-reader reliability versus two-classifier classification accuracy

30 patients have a CT and a pseudo-CT scan image, I want to get them scored for the presence of a disease to know if the pseudo-CT has the same diagnostic performance as CT. I give these two sets (30 ...
141 views

### Low inter-rater reliability for Fleiss Kappa

Context: I used Fleiss Kappa to compute inter-rater reliability for categorical judgements on a list of words. The raters were given instructions and asked to judge whether a list of ~3700 words were ...
38 views

### Concordance Scoring Best & Best Alternative Results

I have a model that outputs two ordered results from a large set: Let's call them the "best" and "next" choices. I set of judges rate this output, and can tell me various things, ...
22 views

### Agreement Statistics for Judgment Task

Situation: I am unsure how many judges I need for good inter-rater reliability and how to calculate the inter-reliability in R depending on the number of judges. Task: I am wanting to obtain people's ...
13 views

### Assessing Consistency and Agreement of Two Groups of Raters in an ordinal dataset

I'm using a real dataset of convenience to look at one group of subjects who are being assessed for "frailty". The goal is to see whether assessments by non-experts are comparable to experts....
• 1
1 vote
175 views

### Disagreement between ICC and paired t-test

I am trying to determine whether the values obtained by an algorithm calculation are systematically different from the the criterion values. The criterion values are obtained via manual observation ...
• 43
40 views

### interrater reliability alternatives to kappa

I have applied a binary classifier to a raster. Within the raster, I have "ground" truth plots where pixels have been manually assigned to one of the two classes. I would like to assess the ...
68 views

### Kappa coefficient for variable number of raters

I want to calculate the Inter-rater Reliability of an annotated dataset, where the number of raters for each subject is variable (3-5 raters) and subjects can be annotated with one of 7 categories. ...
• 1
54 views

### Comparison of experimental method against gold standard?

I have a method X, assumed to be the gold standard. Then, I have a method Y. The response variable (for example, weight) is a continuous variable. For a sample of n individuals, the response variable ...
• 23
31 views

### Best measure of inter coder reliability for single coded qualitative data

I have a table of coded strategies, coded by 3 coders, to determine inter-coder reliability. Coders code by placing a 1 where they believe a strategy fulfills a certain amount of criteria. Green ...
• 1
1 vote
211 views

### Compute inter-rater reliability for categorical ratings with variable raters per target

What is the correct statistic to use for computing inter-rater agreement from data of the following form? ...
• 113
54 views

### Statistics reference for measures of agreement?

I am looking for a statistically-oriented textbook (accessible to one who has knowledge of the material in Casella and Berger's text) which explains the theory behind measures of agreement - e.g., ...
• 3,903
95 views

### How can I use the ICC as a measure for inter-rater agreement on a set of (more or less) correlated items?

I have a question about inter-rater agreement using the ICC. I'm trying to test, whether my study participants (the raters) rate the personality of a fictional character similarly - think of a ...
156 views

### Unable to calculate kappa or weighted kappa when 1+ raters give the same rating to everyone -- what is a good solution?

Are there alternate inter-rater reliability statistical tests one can perform in cases where a rater gives the same rating to everyone? SPSS is unable to return a kappa or weighted kappa value when ...
1 vote
125 views

### inter-rater agreement for data with nominal and ordinal responses

I'm trying to determine inter-rater reliability for ratings produced by 4 unique raters, with four possible ratings, NA, 0, 1, 2 corresponding to a quality judgement. I am interested in the distance ...
• 13
91 views

### Inter-rater reliability for multiple raters for one item (subject)

I have data with 3 raters, each of whom rated 3 items as follows: item a item b item c rater 1 a1 b1 c1 rater 2 a2 b2 c2 rater 3 a3 b3 c3 I successfully calculated the inter-rater reliability for ...