Tagged Questions

0
votes
0answers
34 views

interrater reliability for multiple variables

i and another coder want to assess our agreement. our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
0
votes
0answers
52 views

How to assess Inter-rater reliability of frequency data with two raters using SPSS?

I am doing an observational study on the frequency (number of times) a particular person enters a shop. This study is being carried out by two different observers. I now want to do an inter-rater ...
1
vote
0answers
68 views

What is the proper test for inter-rater reliability (three raters, ordinal data)

I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal ...
7
votes
2answers
218 views

Handling poor inter-rater reliability while minimizing the loss of data

Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
2
votes
0answers
54 views

Assessing the ability of two quantitative methods measuring the same thing

Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
2
votes
1answer
99 views

Basic Inter-rater reliability and type of data

I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability. Research question: Functional Movement Screening: How reliable are we in our ...
3
votes
2answers
189 views

Inconsistency of findings between Kappa and ICC for IRR study

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
2
votes
0answers
73 views

Testing graded response model thresholds for significance?

I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
7
votes
1answer
215 views

What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
19
votes
2answers
485 views

Is Joel Spolsky's “Hunting of the Snark” post valid statistical content analysis?

If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He ...
7
votes
1answer
667 views

How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?

Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
4
votes
1answer
458 views

Interrater reliability in SPSS

I am trying to calculate interrater reliability in SPSS for both pre and post test of the same measure that is administered as part of a prison intake program. The measure has 20 items and then a ...
2
votes
2answers
191 views

Inter-rater reliability of time measurements

I am having some trouble trying to decide what measure of inter-rater reliability to use in a study. Part of a larger study involves accurately determining when participants began (onset) and stopped ...
1
vote
0answers
266 views

How to analyse linked 2x2 tables from multiple raters?

Comment: heavily edited, both to clarify and to fix a mistake. First 7 comments below refer to the previous version. I have an experiment where 3 raters go over the same data 2 times each, trying to ...
3
votes
0answers
198 views

Computing inter-rater reliability in R with variable number of ratings?

Wikipedia suggests that one way to look at inter-rater reliability is to use a random effects model to compute intraclass correlation. The example of intraclass correlation talks about looking at ...
1
vote
1answer
342 views

What inter-rater reliability test is best for continous data with multiple measures?

I am doing a study with one variable with continuous data; the measurement involves measurements done by three people. I wish to do an inter-rater reliability test, e.g. with Bland-Altman analysis, ...
3
votes
1answer
191 views

Inter-rater reliability with many non-overlapping raters

I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
1
vote
0answers
212 views

Kendall's W for multiple objects and multiple surveys

I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios. For example suppose for scenario A we had multiple objects and multiple raters like in the table ...
4
votes
1answer
2k views

Intra- and inter-rater reliability on the same data

I've read some of the answers here (specifically the 'Inter-rater reliability for ordinal or interval data' one). I'm still somewhat perplexed though! I have data for 4 raters, who each rated every ...
4
votes
2answers
842 views

What to do in case of low inter-rater reliability (ICC)?

Background: Eight doctors each rated the same 54 patients on a persuasiveness measure (1-7 Likert scale). The mean score on the persuasiveness measure will eventually be the outcome measure of my ...
4
votes
4answers
229 views

Assessing rater bias where one rater has given one very high rating and the remainder very low ratings

What is a good statistical test to check if there is a bias in judging in a situation that there is one judge that gave extreme scores (high score for one of the contestant and very low scores on the ...
2
votes
1answer
578 views

Assumption of additivity for intra-class correlation

My question concerns the assumption of additivity for intraclass correlation. I shall first explain what I have done and then end with my questions. I want to calculate inter-rater reliability using ...
3
votes
1answer
511 views

Interater reliability

What inter-rate reliability test is best for continuous data? I am doing a study with one variable with continuous data, now the measurement involves measurements done by two people. I would wish to ...
7
votes
2answers
326 views

Interrater reliability for events in a time series with uncertainty about event time

I have multiple independent coders who are trying to identify events in a time series -- in this case, watching video of face-to-face conversation and looking for particular nonverbal behaviors (e.g., ...
3
votes
2answers
264 views

How to “prove” that new measurement tool & process gives same result as old?

I am new to the area of statistics and I am hoping you can suggest methods I may use. Sorry if this is long but I might as well be as clear as possible on my first post :) What I am worried most is ...
3
votes
1answer
6k views

Inter-rater reliability using Intra-class correlation with ratings for multiple objects on multiple properties

Initial Question: I am trying to calculate inter-rater reliability. Previous researchers in this area have used intraclass correlation. SPSS has options for two-way random, mixed random and one-way ...
2
votes
1answer
220 views

Inter-rater reliability between similarity matrices

I have 6 similarity matrices reflecting the proximity of 100 pairs of sounds. Similarity indices are in the range [0,1], with higher values indicating sounds perceived as more alike. What is the most ...
3
votes
2answers
311 views

The best measure of reliability for interval data between 0 and 1 [closed]

I have 6 sets of interval data each of which between 0 and 1. Each set, calculated by a computer program, is related to the degree of similarity between some sounds (pairwise). What do you think in ...
5
votes
2answers
586 views

Repeatability and measurement error from and between observers

I have 3 observers that each take 2 measurements (length and weight) on 100 individuals; these procedures are repeated once (i.e., the same measurements are taken on the same 100 individuals by the ...
12
votes
2answers
7k views

Inter-rater reliability for ordinal or interval data

Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...