I've considered measures like Cohen's kappa (but data is continuous), intra class correlation (reliability, not agreement), standard correlation (will be high when one rater always rates consistently higher than the other rater)... but none seem to represent what I want it to.
I need a measure that will show % agreement between two raters, that have rated videos of multiple subjects in multiple situations and this in 8 different experiments.
So for example, for experiment 1 rater 1 rated 12 subjects in 3 situations and rater 2 did the same, for experiment 2 rater 1 rated 14 subjects in 4 situations and rater 2 did the same. And so on. Which measure would then be adequate to test how much the raters agree over all these 8 experiment? Or just per experiment, so that I can take calculate an average of the 8 values.
I'm sorry if this is an abundant question, but I've never been trained in these kind of statistics and I get completely lost searching the internet for an answer. I need to be sure that this measure is good for my data as I need to use it in my thesis.
Experiment 1, situation 1:
Rater 1 Rater 2 Subj1 13 12 Subj2 85 71 Subj3 67 45 ... ... ...
Experiment 1, situation 2:
Rater 1 Rater 2 Subj1 76 90 Subj2 62 51 Subj3 51 51 ... ... ...
Data looks like this with per experiment a number of subjects that are rated from 0 to 100 (= % freezing) in 3 (or 4) different situations by 2 raters (always the same raters). You can find an example excel file here (might be more clear): dropbox link
Thing is, if rater 1 says 76 and the other says 77, that's good, it's normal that it won't be perfectly the same value, but if it's close, it means that the raters kind of agreed. So continous data, the values go from 0 to 100 and are not categories or anything. If the value by the 2 raters is not exactly the same, that doesn't necessarily mean that there's no agreement: the more distant the values are from each other, means that we had less agreement, the closer they are, the more agreement the raters had.