# Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

298 questions
Filter by
Sorted by
Tagged with
1k views

### Intraclass correlation coefficient: zero and negative

I am trying to calculate reliability between two raters for continuous data. The data is frequency of negative life events for each participant. My data looks like this: Dataset 1 rater 1: 0 0 0 0 0 ...
1k views

### How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
1k views

### Inter-rater reliability measure with multiple categories per item [duplicate]

I have multiple raters providing categorical ratings. What makes this slightly different from other posted questions is that each rater can provide three, unique categorical ratings. Each rater had ...
256 views

### Comparing two sets of limits of agreement (continuous outcome) - R

I am planning an analysis where I will have children and parents rating the child's pain (numerical score). For each child, I will have a child pain score and a parent pain score. The dyads will ...
598 views

### Does Spearman's $r = 0.38$ indicate agreement?

Can a Spearman correlation coefficient of 0.38 for a specific parameter be considered to demonstrate reasonable agreement between two biological cell types, more specifically a cell line and a ...
210 views

### Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
1k views

### Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
2k views

### Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Stats newbie here, please bear with me. I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/...
856 views

### Calculating interrater agreement for multiple choice questions [duplicate]

I'm trying to figure out how to calculate interrater agreement so I can aggregate raters' responses. I'm asking 5 raters to select topics (out of 6 topics: A, B, C, D, E or F) that describe texts ...
74 views

### Inter-raters reliability for sentiment analysis lexicon

I have a list of words (more than 2000) which were rated by 4 raters for sentiments like anger (scale from 1 to 5). I run an alpha test on SPSS which gives me a coefficient of >8 which shows a good ...
43 views

### Which statistical method should I use to compare test results of two groups with each other and with exact values?

I am currently working on an in-vitro medical device for an academy project with my teammates to see if our approach to a certain problem is working. We are hiring two groups of volunteers (15 people ...
246 views

### Calculate p-value for difference between ICC values [duplicate]

I cannot seem to find an answer to the following problem: I am doing interreader correlation studies where I calculate ICC values. It is quite easy to do this in R, but how do I calculate a p-value to ...
6k views

### Matthews correlation coefficient with multi-class

Matthews correlation coefficient ($\textrm{MCC}$) is a measurement to measure the quality of a binary classification ([Wikipedia][1]). $\textrm{MCC}$ formulation is given for binary classification ...
69 views

### How to determine which variable agrees best with a gold standard?

I have a dataset of 100 patients. Each patient has 3 lab tests: Lab Test 0: "Gold Standard" test which will serve as my reference standard Lab Test 1 Lab Test 2 I want to compare the differences ...
804 views

### Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
722 views

### Standard Error of Measurement (SEM) in Inter-rater reliability

I am currently writing my thesis about inter-rater reliability of a diagnostic tool between raters. I want to use the standard error of measurement with the formula: SEM of rater 1 and rater 2 = SD * ...
4k views

### Power analysis for inter-rater reliability study (Kappa) with multiple raters

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of ...
78 views

### Testing for intra-subject agreement across categories

I am looking at whether there is agreement between a subject's standardized weight (light-for-age or heavy-for-age) in early life and adulthood. I have coded light-for-age as '1' and heavy-for-age as '...
191 views

### Visualizing interrater disagreement

I am trying to visualize the process of two raters who have each rated the same dataset. Each row (line in the plot) represent a heart beat (or an error) on an ecg. Every disagreement was discussed ...
124 views

### Inter-rater reliability or another method?

I've asked about the mechanics of Inter-rater reliability before, but I guess I should have asked this question instead: If I have a bunch of raters and individuals (who are being judged on ...
87 views

### Obtain concordance correlation coefficient accounting for errors in data

I have two 1-dimensional data sets that measure the same process by two different methods: $$x=[x_1, x_2, ..., x_N]\; ;\; y=[y_1, y_2, ..., y_N]$$ I need to asses the agreement between these two ...
657 views

### Fisher's exact test vs kappa analysis

I was reading a paper where the authors assessed the association between two different diagnostics tests intended to diagnose the same disease and they performed the analysis with Fisher's exact test. ...
2k views

### How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings (...
472 views

### Calculation for inter-rater reliability where raters don't overlap and different number per candidate?

I want to calculate the degree to which the gymnastics judges agree on balance beam scores, i.e., "inter-rater reliability". However, not all judges judge the same candidates, and the number of judges ...
471 views

### How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR. The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254. Is using Fleiss ...
232 views

### rubrics and analyzing essay scores

I have essay scores from one experiment. I have two raters and each rater separately graded the essays by using rubrics. The rubrics consist of 5 criteria, rating scales (1-5) and "descriptions"(...
259 views

### Inter-annotator agreement in multi-label classification

I wonder what is the best metric to measure inter-annotator agreement in multi-label classification for two annotators?
242 views

### How can I calculate inter-rater agreement for two raters assigning ranks?

I am new to statistics and currently trying to calculate the inter rater agreement between two entities based on their rankings of a set of things. For example if i give person 1 a set of 5 things and ...
328 views

### Which ICC should I use for inter-rater reliability?

In my study a dog's owner and dog's walker each filled out a personality assessment for a target dog. I have 60 dogs in total. There were multiple questions that scored the dog on 5 dimensions. I'm ...
339 views

### Inter-rater reliability for two raters, multiple binary codes

I have two raters who utilized a binary codebook with codes (A, B, C) to code a series of responses. The codes are not independent, and a response could have more than one code (or no codes at all). ...
499 views

### Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
3k views

### Bland-Altman (Tukey Mean-Difference) plot for differing scales

I find that Bland-Altman plots for comparing two methods are extremely useful in assessing agreement. However, I'm curious if there is a similar method or transformation that can be used when the ...
377 views

### Inter-rater agreement of a gold standard dataset - a ceiling for reliable evaluation of algorithms?

In my field, a dated gold standard dataset is used to track progress in algorithm development. Now when the state-of-the-art algorithms obtain higher correlation than is the inter-rater agreement of ...
737 views

### Rater's agreement for each item

I would like to calculate the agreement between 2 or more raters on all the judgments from a set, but also for each judgment. For example: ...
40 views

### Assessing the agreement between two tests

I have genetic data. I have two tests that pick up changes in the genome (amplifications and deletions) at several different points in the DNA. There are three basic outputs- high, normal, low. I want ...
64 views

### Inter-rater reliability: two raters rate dataset A and two raters rate dataset B how to combine it into one number?

Me and one other person (Alice) counted the amount of boys and girls playing on a school playground (consent of parents and school were given of it being videotaped). Two other persons (Bob and Claire)...
66 views

### How to measure correlations/congruence between parent and teacher ratings of children's personality

I was wondering if anyone could help. I am investigating children's personality, and I have collected both parents' and teachers' ratings for children on a 24-item personality scale of the big-five (...
518 views

### Comparing measurements obtained from >2 devices (Bland-Altman, Tukey mean difference plot)

I would like to compare different measurements in regard to agreement. Quite often this is done using a Bland-Altman plot (also called Tukey mean difference plot) in medical literature. What I would ...
244 views

### Am I using Fleiss' kappa coefficient correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
3k views

### Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
368 views

### Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
481 views

### How can I quantify test-retest reliability with three waves of measurement?

I tested subjects and repeated the test 2 times, so I did it 3 times in total. My question is: does anyone know how to evaluate the test-retest reliability? Is there another method (besides test-...
303 views

### Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
698 views

### How is the chance-level confusion matrix calculated?

I applied an ML technique on my dataset, and got this confusion matrix: 0 1 0 162 62 1 27 50 Funnily, the overall accuracy is worse than ...
3k views

### how can i calculate the degree of agreement between two methods

I have a sample of 26 learners and I used two different methods to identify their personalities. My problem is I am not that good in statistics and I want variables that I can use within SPSS to ...
320 views

### How to test the inter-rater reliability of a nominal coding process that generates ordinal data?

A researcher studying hunger on a tropical island asks respondents, "What did you eat for lunch yesterday?" and the possible choices are "(4) Nothing; (3) a banana; (2) an unripe coconut; (1) a ripe ...
48 views

Data: ...
537 views

### Testing two raters' time-point data for interrater reliability [closed]

My data consists of two raters interpreting one specific phenomenon to occur at different points in time (the observations are not paired, the raters actually identified different amounts of ...