# Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

298 questions
Filter by
Sorted by
Tagged with
1k views

### How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
209 views

### advice on a solution attempt: interrater reliability of time-point data [duplicate]

In my data, two coders annotated subjectively (but independently), when certain time-point phenomena (a specific turn in a movement pattern) occurred. The data for the first 14 seconds looks ...
186 views

### How can I calculate inter-rater reliability for continuous measures that are not normally distributed?

My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data ...
68 views

### Correlation between diagnosis codes

How do I run a correlation analysis on medical diagnosis (Dx) codes between two years at patient level. Data runs into 45000+ observations for each patient with different diseases across years and I ...
117 views

### Observer agreement in multilevel binary data

I have a set of participants indicating with drawn circles where they experience pain on a manikin of the human body. These drawings were then independently rated by two human raters: each of 29 body ...
100 views

### Is there more than one way of calculating F1-score

The reviewer of an article that I submitted to a journal commented that I never explicitly state how I calculate the F1-score. Is there more than one way to do it assuming that we are talking ...
172 views

### Comparing angle measurements from two different techniques

I'm working on a research project wherein we have 50 subjects. We have 3 x-rays per patient that all show a slightly different view of the same anatomic region. I am trying to compare an angle ...
890 views

### Inter-rater reliability for unordered categorical data

I have some content analysis data that consist of unordered mutually exclusive categories as rated by two coders. What are useful approaches towards assessing inter-rater reliability for these data? ...
3k views

### Comparing inter-rater agreement between classes of raters

I'm interested in comparing ratings of the same objects that were done by raters from 2 different GROUPS/CLASSES (Experts, and Semi-Experts), so I can decide whether Semi-experts can replace experts ...
779 views

### When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
402 views

### Fleiss' Kappa alternative for inter-rater reliability with multiple answers

I have a dataset where researchers assign a binary yes/no answers to cells in a matrix. For each row more than one, but at least one, 'x' is allowed. For example, researchers 1 may code: ...
54 views

### A measure of correspondence between ranked ordinal data

I would like to find an appropriate way to measure the similarity between two sets of data with the following characteristics: Both sets contain 10 categorical observations. The categories observed ...
548 views

### How can I best deal with the effects of markers with differing levels of generosity in grading student papers?

Around 600 students have a score on an extensive piece of assessment, which can be assumed to have good reliability/validity. The assessment is scored out of 100, and it's a multiple-choice test ...
591 views

### What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
326 views

### Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
190 views

### What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
866 views

### which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
11k views

### Do I run a one-tailed or two-tailed test when running Spearman's correlation between interval and ordinal data?

I have one interval variable on a 1-25 point interval scale, i.e. placement test scores where 25 is the highest competence score (data not normally distributed), and six different ordinal variables on ...
35 views

### Analysis of ordinal data with multiple assessments per unit… the right approach?

Suppose I have a dataset with the following information: N objects, each of which can be rated The number of ratings per object is variable (raters are unidentified), e.g., there are 30 objects each ...
40 views

### Expected agreement if random

I am looking to measure agreement between participants in choosing which member of a group is most like certain attributes. I want to calculate the expected agreement if they were to choose by chance. ...
213 views

### Can you test if one judge gives significantly different scores from other judges?

If I had a small number of observations (about 5) in a sample and the population standard deviation was unknown, what test could be done to see if the value of one of the observations was ...
2k views

### Inter-rater agreement for Likert scale

What is the best inter-rater agreement test for Likert scale type questions? As far as I see, Cronbach's $\alpha$ is for internal consistency, or it shows how good items are related to describe the ...
6k views

### Creating and interpreting Bland-Altman plot

Yesterday I heard of the Bland-Altman plot for the first time. I have to compare two methods of measuring blood pressure, and I need to produce a Bland-Altman plot. I am not sure if I get everything ...
98 views

### Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
1k views

### Measure of accuracy or measure of agreement for continuous data with true values

I have two continuous variables. Variable $X$ represents the true values (basically it is atmospheric pressure measured via a barometer). Variable $Y$ represents a way to approximate the atmospheric ...
1k views

### Agreement statistics for confusion matrix with nominal categories

I have a decision support system that recommends Treatments (T1 to T8) to patients, where T'...
173 views

### How will two raters score videos?

I have 260 teacher conferences to score using a rubric. How do two raters need to score these for rater agreement? Do we need three raters?
200 views

### How to summarize measurement variability for repeated measures?

I have a data set where different measures are taken repeatedly (3) times for multiple samples, like so: A B C X ... Y ... So, for people X, Y, etc I have ...
250 views

### How do I calculate reliability between an algorithm and multiple human experts?

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
635 views

### Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
191 views

### What is the analysis techniques for comparing two sets of ratings

Say, I have collected 50 comments. For each comment, I am planning to get 10 ratings from non-expert raters for the following criteria: "interesting vs. not interesting". Based on the plurality of ...
13k views

### ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
31k views

### Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS. Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important). All ...
2k views

### Kendall's coefficient of concordance (W) for ratings with a lot of ties

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As ...
122 views

### Looking for agreement amongst respondents

I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of ...
136 views

### New method for meta-analysis to determine agreement between multiple sources

I'm writing a paper on a method to determine the statistical significance of whether a sample of sources are in mutual agreement on specific premises. This is done by counting the number of sources in ...
7k views

### Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...