2
$\begingroup$

I've considered measures like Cohen's kappa (but data is continuous), intra class correlation (reliability, not agreement), standard correlation (will be high when one rater always rates consistently higher than the other rater)... but none seem to represent what I want it to.

I need a measure that will show % agreement between two raters, that have rated videos of multiple subjects in multiple situations and this in 8 different experiments.

So for example, for experiment 1 rater 1 rated 12 subjects in 3 situations and rater 2 did the same, for experiment 2 rater 1 rated 14 subjects in 4 situations and rater 2 did the same. And so on. Which measure would then be adequate to test how much the raters agree over all these 8 experiment? Or just per experiment, so that I can take calculate an average of the 8 values.

I'm sorry if this is an abundant question, but I've never been trained in these kind of statistics and I get completely lost searching the internet for an answer. I need to be sure that this measure is good for my data as I need to use it in my thesis.

Experiment 1, situation 1:

    Rater 1    Rater 2
Subj1 13         12

Subj2 85         71

Subj3 67         45
...   ...        ...

Experiment 1, situation 2:

    Rater 1    Rater 2
Subj1 76         90

Subj2 62         51

Subj3 51         51
...   ...        ...

Data looks like this with per experiment a number of subjects that are rated from 0 to 100 (= % freezing) in 3 (or 4) different situations by 2 raters (always the same raters). You can find an example excel file here (might be more clear): dropbox link

Thing is, if rater 1 says 76 and the other says 77, that's good, it's normal that it won't be perfectly the same value, but if it's close, it means that the raters kind of agreed. So continous data, the values go from 0 to 100 and are not categories or anything. If the value by the 2 raters is not exactly the same, that doesn't necessarily mean that there's no agreement: the more distant the values are from each other, means that we had less agreement, the closer they are, the more agreement the raters had.

Improve this question
$\endgroup$
2
  • $\begingroup$ To me, it looks like a categorical data, not continuous data. Can you post a minimal hypothetical reproducible example/data? $\endgroup$
    – Sahil Dahiya
    Commented Jan 2, 2018 at 16:29
  • $\begingroup$ I edited the post to include an example. Thank you for helping. I also included an example excel file via a dropbox link, because it might be more clear that way. $\endgroup$
    – Caeline
    Commented Jan 3, 2018 at 17:51

1 Answer 1

Reset to default
0
$\begingroup$

If the categories are considered predefined (i.e. known before the experiment), you could probably use Cohen's Kappa or another chance-corrected agreement coefficient (e.g. Gwet's AC, Krippendorff's Alpha) and apply appropriate weights to account for partial agreement; see Gwet (2014). However, it seems like an ICC could be appropriate, too. I do not really understand what the "situations" are all about.

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability. Gaithersburg, MD: Advanced Analytics, LLC.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.