Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

sample size advice for matching nominal variables

I have two databases of patients and medications. Each database has the same patients. Both have millions of records. My job is to select a sample from each and confirm that the medications match. ...
0
votes
0answers
93 views

How do the $r^{2}$, amount of variance explained and concordance correlation coefficient relate?

Let us define the $r^{2}$ as: $$ r^{2} = \left(\frac{\sum_{i=1}^{n}(obs_{i} - \overline{obs})(pred_{i}-\overline{pred})}{\sqrt{\sum_{i=1}^{n}(obs_{i} - \overline{obs})^{2}}\sqrt{\sum_{i=1}^{n}(pred_{i}...
1
vote
0answers
22 views

Measure of agreement multiple elements within multiple items

Two raters assessed the risk of bias in 3 randomised controlled trial (RCT) research articles using a standard critical appraisal checklist. The checklist has 10 questions addressing key risks of bias ...
1
vote
0answers
27 views

Calculating interclass correlation but with different observer experience

I have tried finding an answer to the following question both here and using PubMed and google but without luck. I have been asked to calculate the intraclass correlation coefficient from the ...
2
votes
2answers
100 views

Is there a strict relation between Accuracy and Cohen's Kappa (measures of classification quality/agreement)?

Say I have two models (classifiers), $M_1$ and $M_2$, each with their own accuracy w.r.t. the ground truth, and I calculate also Cohen's Kappa between each model and the ground truth (as a measure of ...
1
vote
0answers
23 views

Modelling how test agreement varies as a function of a covariate

I have a group of human subjects who have been given two versions of a test measuring the same outcome. I am interested in the agreement between these two tests, and whether their agreement varies as ...
0
votes
0answers
9 views

What is a good measure of the inter-rater agreement when each annotation is a ranked list of 3 elements and there are more than 2 annotators?

What is a good measure of the inter-rater agreement when one has the following two conditions: each annotation is a ranked list of 3 elements (the  annotator can choose them amongst 10 elements), ...
2
votes
1answer
128 views

Fleiss kappa vs Cohen kappa

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood? When would one use Fleiss kappa over Cohen kappa? What are the ...
0
votes
1answer
25 views

Assessing agreement

Which statistical analysis would you suggest to assess agreement when the scales of the two methods are not identical, but they still measure the same underlying phenomenon? I'm analyzing a new ...
0
votes
0answers
12 views

Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...
1
vote
0answers
22 views

Agreement between two groups of raters

What is the best method for assessing agreement between to groups? I have 2 ICU docs and 2 radiologists evaluating x-ray exams with yes/no outcome. I can use Cohen kappa for agreement within each ...
0
votes
0answers
11 views

Whats the difference between uncorrected and corrected Kendall's W - Kendall's coefficient of concordance? (synchrony R Package)

The synchrony R package has a function to calculate Kendall's coefficient of concordance also known as Kendall's W: kendall.w. The function has two outputs for ...
0
votes
1answer
18 views

Inter-rater agreement with only one subject?

I'm trying to describe inter-rater reliability for a group of 5 researchers working on the same project. We're administering the Hamilton Anxiety rating scale which has 14 items, each item having 4 ...
0
votes
1answer
31 views

Inter-rater reliability or intraclass correlation for one subject

I'm looking for a way, in R, to get measures of agreement for each item. I have 32 ordinal items (we want separate statistics for each), and 66 raters.I have two subjects, but we want to look at the ...
0
votes
0answers
15 views

How do I calculate Inter-rater reliability between one rater and a group of raters?

So let's say I have a study in which I give a categorical rating to bunch of items. Each item could be a pair of images and I have to rate which one is brighter for example. Perhaps three ratings ...
0
votes
0answers
54 views

Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip. The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...
0
votes
0answers
17 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
1
vote
1answer
28 views

Cohen's kappa and confounders

I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee. For example, age, ...
0
votes
2answers
39 views

Chi square test between samples with very different size

I am to estimate the chi square between two samples of size N and M, where in general N << M (e.g. N=100,000 M=10,000,000). In turn, this affects the uncertainty of each sample. I wonder if ...
0
votes
0answers
16 views

Measuring IRR between groups as well as improvement in agreement within groups

I have been going back and forth on proper stats for this question, and I am stumped. Basic design is a 2 x 3 mixed design. Two between-subject groups - both are measured for their opinions at ...
0
votes
0answers
12 views

Inter-Rater Reliability for Relative Strength of a Measure

Let's start with a classic: You're trying to get 3 judges to measure the creativity of different art projects, from 1-10, with 1 being 'Uninspired' and 10 being 'Incredibly Creative'. You want to ...
2
votes
1answer
90 views

Inter-rater agreement for non-fully crossed designs

I have a set of n subjects to be rated by i raters on a categorical scale that distinguishes k classes. Both the subjects and the raters are randomly sampled from a larger population of subject and ...
3
votes
1answer
52 views

Inter-rater agreement: number of raters vs. number of test subjects

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical ...
0
votes
1answer
39 views

Krippendorff's alpha (and other IRR stats) giving counter-intuitive results

I am asking 1 question to 7 raters. They reply with one of three possible nominal values; for the sake of this example, let's call them 1, 2, and 3. I need a measure of inter-rater reliability that ...
0
votes
0answers
16 views

How to Compute Krippendorff's Alpha for Text Continua?

I have a question about the inter-annotator reliability measure of Krippendorff's alpha for text continua or text unitization. The measure is explained in detail in the paper. I have texts and 2 ...
0
votes
0answers
29 views

Krippendorff's Alpha for unitizing texts

I have a question about using alpha to measure Krippendorff's inter-annotator reliability in qualitative data. After reading the papers, I'm still not sure how to handle this case. I'm working on a ...
2
votes
0answers
129 views

Krippendorff's alpha in R for multi-label annotation

I need to calculate inter-rater agreement (2 annotators) on a small dataset (let's say 10 items), where annotators used between 1..4 labels per item. The total number of different labels is pretty ...
1
vote
1answer
225 views

nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)?

I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M ...
3
votes
1answer
335 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
1
vote
1answer
33 views

Is it appropriate to use a confusion matrix for frequency data?

I am testing the accuracy of a machine learning approach that counts cars in images. I have both a predicted dataset and a "real" dataset that was generated by a human. For example, this is what my ...
2
votes
0answers
35 views

Applying inter-rater reliability principles

Background: I want to perform a survey/questionnaire for a group of people with very different background and knowledge. Some of them are environmental scientists, anthropologists, and climate ...
0
votes
0answers
29 views

Krippendorff's alpha getting low after restricting data

I am calculating inter-rater agreement for ~350 raters, each of who rated 100 items using Likert scale. The original scale has range [-3,3], using this python implementation, I got $\alpha=.35$, which ...
0
votes
0answers
11 views

Inter-rater reliability many coders but only one case

I have a specific performance assessment metric which I aim to get some reliability and validity data for. And I really would appreciate some help for the reliability: Background In my research, ...
0
votes
0answers
78 views

Bootstrapping non-random samples

I perform some analysis (with the goal to select some relevant variables that are related between two experiments) on a group of samples (n = 158) but I would like to know how robust are, before going ...
2
votes
0answers
9 views

estimation inter rater reliability for strings of characters (i.e. URLs)

I have multiple raters extracting URLs from the internet based on search terms. The core issue is that a URL amounts to a string wherein two raters might come to the same URL but one string is a ...
1
vote
0answers
13 views

Demonstrating improved agreement on clinical decision making using the Kappa coefficient

I have a large number of respiratory doctors - some from University hospitals and some from community non-academic hospitals. They have been asked to evaluate 60 patients with various different lung ...
1
vote
0answers
35 views

Comparing 3 raster maps of continuous surface data in R

I have 3 maps representing spatial distribution in hunting pressure. These maps were derived using different methods and I am now interested in comparing them to assess how they might differ/agree ...
1
vote
1answer
43 views

Are there any statistical tests or measures for paired data with more than two groups?

Are there any statistical tests or measures for paired data with more than two groups? My dependent variable is pretty flexible, but I'd really like to have five independent variables compared ...
0
votes
0answers
13 views

Measuring inter-rater reliability in a manual clustering task

There are plenty of methods for measuring inter-rater reliability in tasks involving rankings, classifications, and labeling (joint probability, Kappa statistics, etc.) There doesn't however seem to ...
0
votes
0answers
10 views

Best way to measure reliability of auto-correlated data

I'm trying to compare different ambulatory devices for measuring heart rate against a gold standard in-lab assessment. What is the best way to determine whether a device is worth using. I can use ...
3
votes
1answer
29 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
1
vote
1answer
98 views

interrater-reliability with likert-scale and different raters

I am collecting data from a survey that includes a likert scale of 5 points. Each rater answers 18 questions (probably different questions). For each question I will have answers from 10 different ...
0
votes
0answers
18 views

intra rater variability

150 raters from 30 different centres inside 10 geographical region (so 3 centres from 10 regions and raters from each centre) grade 10 images twice on an ordinal scale. How can I calculate intra ...
0
votes
0answers
378 views

Corelation between two dichotomous variables

I have a data from a research with responses (yes/no) about three questions (A,B,C) and I would to see if the variables are correlated. First of all i compared the proportion of agreement(yes-yes, no-...
1
vote
0answers
30 views

Help analyzing data ranked on a scale

So, I thought it would be interesting to try an experiment with my family. I baked 13 different kinds cookies this year, and then told my 8 family members to rank from each cookie from 1-13. Is there ...
0
votes
0answers
614 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
3
votes
3answers
82 views

How to compare different clusterings?

In an 'unguided' experiment, we asked several people to make groups with a defined number of objects presented to them. They were free to do as many classes as they wanted, and to choose their names, ...
0
votes
0answers
61 views

How to compare agreement of two methods with gold standard?

Let's assume that there are two medical students who both guess the age of several patients. If the actual age of the patients is the gold standard, I can assess the agreement of each student with the ...
1
vote
0answers
20 views

Comparing inter-rater agreement between classes of raters with only two ratings per object

In order to decide whether semi-experts can replace experts in my setting, I am interested in comparing ratings of the same objects that were done by two raters from 2 different groups (semi-expert, ...
0
votes
0answers
47 views

Bayesian agreement

Assume that there are k expert measuring some quality x with a number between zero and one, $x_i \in [0,1], i=1,2,...,k$. I would like to first know the aggregated quality of x with respect to the ...