# Tagged Questions

72 views

### Sample size for multiple rater inter-rater reliability study

How do I calculate the minimum overall sample size required for a study using multiple raters (n=5) assessing inter-rater reliability using Fleiss generalized kappa for binary variables, Kendall's W ...
83 views

### interrater reliability for multiple variables

i and another coder want to assess our agreement. our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
92 views

### How to assess Inter-rater reliability of frequency data with two raters using SPSS?

I am doing an observational study on the frequency (number of times) a particular person enters a shop. This study is being carried out by two different observers. I now want to do an inter-rater ...
96 views

### How to assess relationship between responses to multiple likert items between two raters when there are multiple rater pairs?

I have two data sets in which the same Likert scale questions were answered by two different responders. I want to calculate which level they are correlated. My data set is like ...
91 views

### What is the proper test for inter-rater reliability (three raters, ordinal data)

I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal ...
271 views

### Handling poor inter-rater reliability while minimizing the loss of data

Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
70 views

### Assessing the ability of two quantitative methods measuring the same thing

Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
151 views

### Basic Inter-rater reliability and type of data

I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability. Research question: Functional Movement Screening: How reliable are we in our ...
313 views

### Inconsistency of findings between Kappa and ICC for IRR study

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
87 views

### Testing graded response model thresholds for significance?

I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
248 views

### What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
537 views

### Is Joel Spolsky's “Hunting of the Snark” post valid statistical content analysis?

If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He ...
782 views

### How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?

Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
560 views

### Interrater reliability in SPSS

I am trying to calculate interrater reliability in SPSS for both pre and post test of the same measure that is administered as part of a prison intake program. The measure has 20 items and then a ...
220 views

### Inter-rater reliability of time measurements

I am having some trouble trying to decide what measure of inter-rater reliability to use in a study. Part of a larger study involves accurately determining when participants began (onset) and stopped ...
291 views

### How to analyse linked 2x2 tables from multiple raters?

Comment: heavily edited, both to clarify and to fix a mistake. First 7 comments below refer to the previous version. I have an experiment where 3 raters go over the same data 2 times each, trying to ...
239 views

### Computing inter-rater reliability in R with variable number of ratings?

Wikipedia suggests that one way to look at inter-rater reliability is to use a random effects model to compute intraclass correlation. The example of intraclass correlation talks about looking at ...
590 views

### What inter-rater reliability test is best for continous data with multiple measures?

I am doing a study with one variable with continuous data; the measurement involves measurements done by three people. I wish to do an inter-rater reliability test, e.g. with Bland-Altman analysis, ...
229 views

### Inter-rater reliability with many non-overlapping raters

I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
226 views

### Kendall's W for multiple objects and multiple surveys

I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios. For example suppose for scenario A we had multiple objects and multiple raters like in the table ...
2k views

### Intra- and inter-rater reliability on the same data

I've read some of the answers here (specifically the 'Inter-rater reliability for ordinal or interval data' one). I'm still somewhat perplexed though! I have data for 4 raters, who each rated every ...
1k views

### What to do in case of low inter-rater reliability (ICC)?

Background: Eight doctors each rated the same 54 patients on a persuasiveness measure (1-7 Likert scale). The mean score on the persuasiveness measure will eventually be the outcome measure of my ...
269 views

### Assessing rater bias where one rater has given one very high rating and the remainder very low ratings

What is a good statistical test to check if there is a bias in judging in a situation that there is one judge that gave extreme scores (high score for one of the contestant and very low scores on the ...
647 views

### Assumption of additivity for intra-class correlation

My question concerns the assumption of additivity for intraclass correlation. I shall first explain what I have done and then end with my questions. I want to calculate inter-rater reliability using ...
590 views

### Interater reliability

What inter-rate reliability test is best for continuous data? I am doing a study with one variable with continuous data, now the measurement involves measurements done by two people. I would wish to ...
370 views

### Interrater reliability for events in a time series with uncertainty about event time

I have multiple independent coders who are trying to identify events in a time series -- in this case, watching video of face-to-face conversation and looking for particular nonverbal behaviors (e.g., ...
269 views

### How to “prove” that new measurement tool & process gives same result as old?

I am new to the area of statistics and I am hoping you can suggest methods I may use. Sorry if this is long but I might as well be as clear as possible on my first post :) What I am worried most is ...
7k views

### Inter-rater reliability using Intra-class correlation with ratings for multiple objects on multiple properties

Initial Question: I am trying to calculate inter-rater reliability. Previous researchers in this area have used intraclass correlation. SPSS has options for two-way random, mixed random and one-way ...
246 views

### Inter-rater reliability between similarity matrices

I have 6 similarity matrices reflecting the proximity of 100 pairs of sounds. Similarity indices are in the range [0,1], with higher values indicating sounds perceived as more alike. What is the most ...
321 views

### The best measure of reliability for interval data between 0 and 1 [closed]

I have 6 sets of interval data each of which between 0 and 1. Each set, calculated by a computer program, is related to the degree of similarity between some sounds (pairwise). What do you think in ...