Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
2 votes
1 answer
41 views

assessing which random sample agrees more with a preferred ranking

I am not a mathematician or a statistician. But, I think the question I have is related to statistics. I will start with a made up example. If I can grade apples into,say four grades from 1 to 4, one ...
user avatar
  • 23
0 votes
0 answers
16 views

Can there be a single limit of agreement value in Bland-Altman analysis?

While trying to learn Bland-Altman analysis, I came across a situation that confused me a bit. I can understand upper (mean + 2 SD) and lower limits (mean – 2 SD), but a single limit of agreement (LOA)...
user avatar
  • 51
1 vote
1 answer
29 views

How to determine agreement between clustering methods?

Let's say you want to compare the outcome of KMeans and KMedoids. How to determine if cluster 1 from KMeans can be compared with cluster 1 with KMedoids. Or, in other words, let's say KMeans labels ...
user avatar
1 vote
0 answers
36 views

Can I compare Cohen's and Fleiss' kappa values?

I am running an experiment where a number of human specialists perform a task by hand, and then a computer algorithm performs the same task. I want to determine how similar the algorithm's results are ...
user avatar
  • 111
0 votes
0 answers
16 views

Best statistical test and power calculation for check of interpretation

What is the best statistical test for the following situation. A radiologist interprets chest X-rays (CXRs) of a population. His read of the CXRs is either Yes there is disease or No there is not ...
user avatar
0 votes
0 answers
29 views

Inter-rater reliability for 2 coders of 30 responses each for 100 different items

I have a data set with the structure outlined below which was coded by two independent raters and I want to determine inter-rater reliability. The dataset contains responses to 100 different items by ...
user avatar
  • 1
0 votes
1 answer
21 views

Calculate interrater reliability for multiple trials and raters in R

I have data for a large number of Trials (only three shown here) and ratings by subjects A, B...
user avatar
1 vote
2 answers
45 views

Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
user avatar
  • 13
0 votes
1 answer
16 views

When is it more appropriate to use linear or quadratic weights to measure inter-rater reliability (Fleiss's kappa or Gwet's AC1?) [duplicate]

I am analysing the results of a linguistic acceptability judgment study where 36 speakers had to rate a finite number of sentences on a 7-point Likert scale (so an ordinal scale, although I am not ...
user avatar
0 votes
0 answers
43 views

Using and Reporting ICC for Aggregated Data About Groups of People

I've designed a survey that consists of multiple questionnaires with Likert-scale questions, each of which targets a specific variable in my research model. The participants of this survey are ...
user avatar
0 votes
1 answer
53 views

Inter annotator agreement when the number of annotators is not consistent across the samples [duplicate]

I have a sample of around 100 that has been annotated as either 0, 0,5, or 1 by 5 annotators. For some reason, I found that some of the annotations need to be deleted. Thus, for some samples, we have ...
user avatar
  • 135
0 votes
0 answers
25 views

appropriate ICC

My apologies this is a rookie question, I am aware of what ICC(Intraclass correlation) is, and that it measures how strongly subjects in the same group resemble each other. However I am not sure how ...
user avatar
1 vote
0 answers
26 views

Quantifying agreement of ordinally ranked sequences

I've been using a very particular set of closely related metrics to quantify how well two ranked sequences agree with one another. I'd like to know if this way of thinking is well-known and if any of ...
user avatar
0 votes
0 answers
18 views

interrater reliability with 12 pairs parents rating their own children in categorical variables

Please accept my apologies if the answer to this question already exists. This is the first time for me to do a study investigating the interrater reliability of a questionnaire. There are 12 pairs of ...
user avatar
  • 1
0 votes
1 answer
42 views

Inter Annotator Agreement for Question Answering

I would like to run an Inter Annotator Agreement (IAA) test for Question Answering. I've tried to look for a method to do it, but I wasn't able to get exactly what I need. I've read that there are ...
user avatar
0 votes
0 answers
7 views

Survey analysis and data handling

I have survey responses from 67 raters. They responded to 7 clinical scenarios. Each scenario had a set of possible responses which are not numeric and not ordinal. A response can be one of 10 ...
user avatar
0 votes
0 answers
27 views

What is the correct inter-rater reliability measure for 2 raters annotating video frames with one of 3 categories?

In my problem, two raters are given 30 videos. For the videos each second will be annotated with one of 3 categories (A making an error, B making an error, no error). For instance, a video could be ...
user avatar
0 votes
0 answers
13 views

Pearson Correlation for Inter and Intra-reliability coeffciient

I am reading this research paper entitled, "Inter-Rater Reliability and Intra-Rater Reliability of Assessing the 2-Minute Push-Up Test" since it is somewhat similar to what I am currently ...
user avatar
0 votes
1 answer
25 views

Measuring within-group similarity for ranked list responses

Individuals are completing a task in which they rank 15 items (1-15 according to importance). These individuals will later be working in groups of 4, so I am interested in measuring the similarity of ...
user avatar
  • 1
0 votes
0 answers
30 views

Inter-annotator agreemnet on a dataset with no annotator id

I have a dataset of annotations like the following, ...
user avatar
0 votes
0 answers
14 views

Proper metric to compare continuous to discrete data? [duplicate]

I have two measurements of the same thing. The first is a subjective rating score, the second is a quantitative metric. The rating score is discrete (0-10), while the quantitative metric is any value ...
user avatar
0 votes
0 answers
15 views

How to compare scale ratings of multiple dimensions from two external raters and several participants who provide different numbers of responses?

I started with Krippendorff's Alpha but if I understand correctly, that assumes that ratings by participants are all by the same individual? Nearly 500 participants produced short pieces of writing ...
user avatar
0 votes
0 answers
14 views

Test-retest reliability with varying number of measurements

I developed a new method to measure a certain quantity on a radiological image of a patient. I have available varying numbers of studies per patient. I applied the method to all images available for ...
user avatar
0 votes
0 answers
9 views

Kappa Agreement analysis for two related classifications

We conducted an experiment where participants compare a metric map and sketch map of the same area. In the sketch map, they are asked to classify generalized (yellow) and non-generalized (green)...
user avatar
  • 1
1 vote
0 answers
47 views

An a posteriori test to determine non difference across >3 groups given a skewed continuous variable of interest, as an alternative to a Friedman-test

Given a set of items of which each item is once measured using a computative model, each time under a different condition, what is the best way to quantify the magnitude of difference between ...
user avatar
1 vote
0 answers
90 views

How to measure machine-human agreement in the light of human-human agreement

Suppose you have a set of subjects, and for each one a measurement of a certain property by a machine and measurements of the same property by several separate humans. Each measurement is a single ...
user avatar
1 vote
0 answers
60 views

What is the correct nesting structure for calculating the Inter-rater reliability or intra class correlation (ICC) with multiple raters?

I need to figure out which inter-rater reliability for multiple raters I should calculate. I'm trying to figure out how I should nest or structure certain observations within my dataset in order to ...
user avatar
0 votes
1 answer
79 views

Checking the relation between two questions

I have set of answers obtained from different respondents to the following questions (with available answers): Q1 --> Do you think this practice is bad? A1 --> YES or NO. Q2 --> What is the ...
user avatar
0 votes
0 answers
115 views

Alternative to Intra-class correlation coefficient (ICC) for continuous data

I am working on a research project that involves a reliability study. In this case, 3 raters have assigned two sets of scores to 50 items: an ordinal score on a scale from 1 - 3, and a continuous ...
user avatar
0 votes
0 answers
26 views

What measure of agreeement should I use for time series data?

I am trying to compare the agreement between two measurements. Each measurement tells me when a device is worn or not worn (binary) and is sampled at the same frequency (1Hz). The measurements were ...
user avatar
1 vote
0 answers
21 views

Calculating the quality of text parsing script

I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research. The ...
user avatar
  • 11
1 vote
1 answer
45 views

Weighted Kappa with rating variables that have different set of categories

I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT Rater1 rated the the ...
user avatar
  • 467
0 votes
1 answer
21 views

Two raters’ inter-reader reliability versus two-classifier classification accuracy

30 patients have a CT and a pseudo-CT scan image, I want to get them scored for the presence of a disease to know if the pseudo-CT has the same diagnostic performance as CT. I give these two sets (30 ...
user avatar
0 votes
1 answer
141 views

Low inter-rater reliability for Fleiss Kappa

Context: I used Fleiss Kappa to compute inter-rater reliability for categorical judgements on a list of words. The raters were given instructions and asked to judge whether a list of ~3700 words were ...
user avatar
  • 1
0 votes
0 answers
38 views

Concordance Scoring Best & Best Alternative Results

I have a model that outputs two ordered results from a large set: Let's call them the "best" and "next" choices. I set of judges rate this output, and can tell me various things, ...
user avatar
0 votes
0 answers
22 views

Agreement Statistics for Judgment Task

Situation: I am unsure how many judges I need for good inter-rater reliability and how to calculate the inter-reliability in R depending on the number of judges. Task: I am wanting to obtain people's ...
user avatar
  • 1
0 votes
0 answers
13 views

Assessing Consistency and Agreement of Two Groups of Raters in an ordinal dataset

I'm using a real dataset of convenience to look at one group of subjects who are being assessed for "frailty". The goal is to see whether assessments by non-experts are comparable to experts....
user avatar
  • 1
1 vote
1 answer
175 views

Disagreement between ICC and paired t-test

I am trying to determine whether the values obtained by an algorithm calculation are systematically different from the the criterion values. The criterion values are obtained via manual observation ...
user avatar
  • 43
0 votes
0 answers
40 views

interrater reliability alternatives to kappa

I have applied a binary classifier to a raster. Within the raster, I have "ground" truth plots where pixels have been manually assigned to one of the two classes. I would like to assess the ...
user avatar
0 votes
0 answers
68 views

Kappa coefficient for variable number of raters

I want to calculate the Inter-rater Reliability of an annotated dataset, where the number of raters for each subject is variable (3-5 raters) and subjects can be annotated with one of 7 categories. ...
user avatar
  • 1
0 votes
0 answers
54 views

Comparison of experimental method against gold standard?

I have a method X, assumed to be the gold standard. Then, I have a method Y. The response variable (for example, weight) is a continuous variable. For a sample of n individuals, the response variable ...
user avatar
  • 23
0 votes
0 answers
31 views

Best measure of inter coder reliability for single coded qualitative data

I have a table of coded strategies, coded by 3 coders, to determine inter-coder reliability. Coders code by placing a 1 where they believe a strategy fulfills a certain amount of criteria. Green ...
user avatar
  • 1
1 vote
1 answer
211 views

Compute inter-rater reliability for categorical ratings with variable raters per target

What is the correct statistic to use for computing inter-rater agreement from data of the following form? ...
user avatar
  • 113
2 votes
2 answers
54 views

Statistics reference for measures of agreement?

I am looking for a statistically-oriented textbook (accessible to one who has knowledge of the material in Casella and Berger's text) which explains the theory behind measures of agreement - e.g., ...
user avatar
  • 3,903
0 votes
1 answer
95 views

How can I use the ICC as a measure for inter-rater agreement on a set of (more or less) correlated items?

I have a question about inter-rater agreement using the ICC. I'm trying to test, whether my study participants (the raters) rate the personality of a fictional character similarly - think of a ...
user avatar
0 votes
1 answer
156 views

Unable to calculate kappa or weighted kappa when 1+ raters give the same rating to everyone -- what is a good solution?

Are there alternate inter-rater reliability statistical tests one can perform in cases where a rater gives the same rating to everyone? SPSS is unable to return a kappa or weighted kappa value when ...
user avatar
1 vote
1 answer
125 views

inter-rater agreement for data with nominal and ordinal responses

I'm trying to determine inter-rater reliability for ratings produced by 4 unique raters, with four possible ratings, NA, 0, 1, 2 corresponding to a quality judgement. I am interested in the distance ...
user avatar
  • 13
0 votes
1 answer
91 views

Inter-rater reliability for multiple raters for one item (subject)

I have data with 3 raters, each of whom rated 3 items as follows: item a item b item c rater 1 a1 b1 c1 rater 2 a2 b2 c2 rater 3 a3 b3 c3 I successfully calculated the inter-rater reliability for ...
user avatar
2 votes
1 answer
32 views

Approaches to visualize difference of two raters from ground-truth

I have a dataset of 80 samples, with each containing two measurements on a 0-10 ordinal scale (Rating_1 and Rating_2) as well as ...
user avatar
0 votes
1 answer
39 views

What's a good way to calculate agreement between examiners when the measure is binary?

I want to assess ability to identify tumor types in humans vs. a computers. There are 7 tumor types. I am interested in calculating agreement between the human examiners but I am not sure the best way ...
user avatar
  • 63

1
2 3 4 5
8