Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
0
votes
1answer
405 views

Which measure for inter-rater agreement for continuous data of 2 raters about multiple subjects in multiple situations?

I've considered measures like Cohen's kappa (but data is continuous), intra class correlation (reliability, not agreement), standard correlation (will be high when one rater always rates consistently ...
2
votes
0answers
11 views

Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
4
votes
2answers
59 views

Quantifying agreement between datasets from different methods of measurement

I have two datasets, A and B. Both ultimately come from the same source, both have the same data attributes, but the datasets use different measurement methods to derive the attributes. We want to ...
0
votes
1answer
22 views

Which is the best coefficient to compute a “user agreement” metric?

Let us suppose that N users are asked to answer yes/no to M questions. For example: ...
2
votes
1answer
523 views

Cohen's Kappa: is it valid to average kappa for different rater pairs across multiple trials?

I'm dealing with the output of a study where raters were semi-randomly paired (based on availability) to rate a total of 50 unique but similar scenarios. Each scenario was rated by a different pair of ...
0
votes
0answers
10 views

Bland-Altman analysis for repeated measure across time with one method of measurement

Just like to seek some clarification. As far as I understand and have read "The Bland–Altman method calculates the mean difference between two methods of measurement (the ‘bias’), and 95% limits of ...
1
vote
1answer
38 views

Fleiss' Kappa v. Cohen's Kappa v. Cronbach's Alpha

I am working on a project that compares two methods of assessing whether certain topics come up in a therapy session. I am comparing a checklist full of dichotomous (y/n) items completed by an ...
0
votes
1answer
95 views

Agreement between methods with multiple observations per individual

I am having trouble interpreting agreement analysis in R. I have a dataset similar to that below with multiple paired observations per subject: ...
1
vote
0answers
255 views

Comparing two sets of limits of agreement (continuous outcome) - R

I am planning an analysis where I will have children and parents rating the child's pain (numerical score). For each child, I will have a child pain score and a parent pain score. The dyads will ...
2
votes
0answers
518 views

Comparing measurements obtained from >2 devices (Bland-Altman, Tukey mean difference plot)

I would like to compare different measurements in regard to agreement. Quite often this is done using a Bland-Altman plot (also called Tukey mean difference plot) in medical literature. What I would ...
1
vote
0answers
177 views

Can I use Bland–Altman plots?

Bland Altman (B.A.) method asses the agreement between two measurement methods and can evaluate the mean difference (bias) of one method against the "gold standard method." With the B.A. method we ...
0
votes
0answers
15 views

sample size advice for matching nominal variables

I have two databases of patients and medications. Each database has the same patients. Both have millions of records. My job is to select a sample from each and confirm that the medications match. ...
2
votes
2answers
115 views

Is there a strict relation between Accuracy and Cohen's Kappa (measures of classification quality/agreement)?

Say I have two models (classifiers), $M_1$ and $M_2$, each with their own accuracy w.r.t. the ground truth, and I calculate also Cohen's Kappa between each model and the ground truth (as a measure of ...
3
votes
1answer
721 views

Standard Error of Measurement (SEM) in Inter-rater reliability

I am currently writing my thesis about inter-rater reliability of a diagnostic tool between raters. I want to use the standard error of measurement with the formula: SEM of rater 1 and rater 2 = SD * ...
0
votes
0answers
103 views

How do the $r^{2}$, amount of variance explained and concordance correlation coefficient relate?

Let us define the $r^{2}$ as: $$ r^{2} = \left(\frac{\sum_{i=1}^{n}(obs_{i} - \overline{obs})(pred_{i}-\overline{pred})}{\sqrt{\sum_{i=1}^{n}(obs_{i} - \overline{obs})^{2}}\sqrt{\sum_{i=1}^{n}(pred_{i}...
1
vote
0answers
23 views

Measure of agreement multiple elements within multiple items

Two raters assessed the risk of bias in 3 randomised controlled trial (RCT) research articles using a standard critical appraisal checklist. The checklist has 10 questions addressing key risks of bias ...
1
vote
0answers
29 views

Calculating interclass correlation but with different observer experience

I have tried finding an answer to the following question both here and using PubMed and google but without luck. I have been asked to calculate the intraclass correlation coefficient from the ...
1
vote
0answers
24 views

Modelling how test agreement varies as a function of a covariate

I have a group of human subjects who have been given two versions of a test measuring the same outcome. I am interested in the agreement between these two tests, and whether their agreement varies as ...
0
votes
0answers
10 views

What is a good measure of the inter-rater agreement when each annotation is a ranked list of 3 elements and there are more than 2 annotators?

What is a good measure of the inter-rater agreement when one has the following two conditions: each annotation is a ranked list of 3 elements (the  annotator can choose them amongst 10 elements), ...
0
votes
1answer
208 views

Which inter-rater reliability test is best for binomial, categorial and continuous variables?

I have two observers who both looked at several scans we performed in patients. They both looked at all scans, they were blinded to clinical data and each others measurements, and they scored several ...
2
votes
1answer
225 views

Fleiss kappa vs Cohen kappa

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood? When would one use Fleiss kappa over Cohen kappa? What are the ...
0
votes
1answer
26 views

Assessing agreement

Which statistical analysis would you suggest to assess agreement when the scales of the two methods are not identical, but they still measure the same underlying phenomenon? I'm analyzing a new ...
0
votes
1answer
199 views

Can you run intraclass-correlations with different raters, and different numbers of raters per participant?

I'm trying to run an intraclass-correlation (inter-rater agreement) for personality data I have collected. However, I work with animals and as such the data has been collected over a period of a year ...
2
votes
0answers
137 views

Krippendorff's alpha in R for multi-label annotation

I need to calculate inter-rater agreement (2 annotators) on a small dataset (let's say 10 items), where annotators used between 1..4 labels per item. The total number of different labels is pretty ...
7
votes
3answers
3k views

Bland-Altman (Tukey Mean-Difference) plot for differing scales

I find that Bland-Altman plots for comparing two methods are extremely useful in assessing agreement. However, I'm curious if there is a similar method or transformation that can be used when the ...
1
vote
2answers
336 views

Correlation coefficient for categorical data resulting from multiple judges

I have data resulting from 3 (fixed) judges observing 15 medical students performing physical exam maneuvers. I'm trying to decide the correct correlation statistic to use. The results can be: done ...
0
votes
0answers
12 views

Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...
1
vote
1answer
62 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...
2
votes
1answer
320 views

How to test the inter-rater reliability of a nominal coding process that generates ordinal data?

A researcher studying hunger on a tropical island asks respondents, "What did you eat for lunch yesterday?" and the possible choices are "(4) Nothing; (3) a banana; (2) an unripe coconut; (1) a ripe ...
0
votes
1answer
429 views

Inter-rater reliability for multiple categorial variables and two raters

We want to know the Inter-rater reliability for multiple variables. We are two raters. The variables are all categorial. This is just an example: ...
1
vote
0answers
22 views

Agreement between two groups of raters

What is the best method for assessing agreement between to groups? I have 2 ICU docs and 2 radiologists evaluating x-ray exams with yes/no outcome. I can use Cohen kappa for agreement within each ...
0
votes
1answer
274 views

Inter-rater reliability simultaneously on multiple binary data

I have drug prescription records on 5 different drugs (A,B,C,D,E) on 2 different time points over a bunch of patients. 1/0 stands for use or not-use the specific drug, respectively. My questions are,...
2
votes
1answer
101 views

Inter-rater agreement for non-fully crossed designs

I have a set of n subjects to be rated by i raters on a categorical scale that distinguishes k classes. Both the subjects and the raters are randomly sampled from a larger population of subject and ...
3
votes
1answer
58 views

Inter-rater agreement: number of raters vs. number of test subjects

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical ...
0
votes
1answer
42 views

Krippendorff's alpha (and other IRR stats) giving counter-intuitive results

I am asking 1 question to 7 raters. They reply with one of three possible nominal values; for the sake of this example, let's call them 1, 2, and 3. I need a measure of inter-rater reliability that ...
0
votes
1answer
37 views

Inter-rater reliability or intraclass correlation for one subject

I'm looking for a way, in R, to get measures of agreement for each item. I have 32 ordinal items (we want separate statistics for each), and 66 raters.I have two subjects, but we want to look at the ...
0
votes
1answer
29 views

Inter-rater agreement with only one subject?

I'm trying to describe inter-rater reliability for a group of 5 researchers working on the same project. We're administering the Hamilton Anxiety rating scale which has 14 items, each item having 4 ...
0
votes
0answers
16 views

Whats the difference between uncorrected and corrected Kendall's W - Kendall's coefficient of concordance? (synchrony R Package)

The synchrony R package has a function to calculate Kendall's coefficient of concordance also known as Kendall's W: kendall.w. The function has two outputs for ...
1
vote
1answer
259 views

nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)?

I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M ...
0
votes
0answers
68 views

How to compare agreement of two methods with gold standard?

Let's assume that there are two medical students who both guess the age of several patients. If the actual age of the patients is the gold standard, I can assess the agreement of each student with the ...
0
votes
0answers
17 views

How do I calculate Inter-rater reliability between one rater and a group of raters?

So let's say I have a study in which I give a categorical rating to bunch of items. Each item could be a pair of images and I have to rate which one is brighter for example. Perhaps three ratings ...
3
votes
1answer
186 views

How can I calculate inter-rater reliability for continuous measures that are not normally distributed?

My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data ...
0
votes
0answers
16 views

How to analyze repeated measures data that has been subsequently rated by coders

My question is this: if you have independent coders rate statements that subjects in the original experiment gave, and there were two statements per subject, and you wanted to examine the difference ...
8
votes
4answers
6k views

Matthews correlation coefficient with multi-class

Matthews correlation coefficient ($\textrm{MCC}$) is a measurement to measure the quality of a binary classification ([Wikipedia][1]). $\textrm{MCC}$ formulation is given for binary classification ...
0
votes
1answer
864 views

Compute Cohen's kappa in multi-label classification

We have 2 different interpretations for our data. Here is an example: ...
3
votes
2answers
537 views

Testing two raters' time-point data for interrater reliability [closed]

My data consists of two raters interpreting one specific phenomenon to occur at different points in time (the observations are not paired, the raters actually identified different amounts of ...
0
votes
0answers
60 views

Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip. The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...
0
votes
0answers
19 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
1
vote
1answer
33 views

Cohen's kappa and confounders

I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee. For example, age, ...
0
votes
1answer
307 views

1 2 3 4 5 6