I am trying to figure out what would be the best statistic to use to quantify the amount of agreement that exists between subjects who were asked to press a button whenever they felt a certain emotion while listening to a short (2min) piece of music. Plotted as time series, the raw data looks something like this (one subject per line; each event(keypress) represented by a blue dot): events defined by subjects

I initially thought of Kendall's tau, but I believe that is only suited to rank data, which is not what I have here. I guess Kendall's W or Cronbach's alpha would be appropriate, although I am not sure whether these are robust to my data having unequal numbers of total events that the different subjects define, e.g. in the screenshot, the first subject defined 3 events, the second subject only one event, etc.

I think correlation would be unsuitable here, as I want to obtain the agreement between all subjects as opposed to between pairs of subjects.

I am also unsure whether what I need to quantify is in fact inter-rater agreement vs inter-rater reliability (e.g. see Liao et al 2010).

What I've described so far aims to obtain a single number that expresses the overall agreement between subjects across the entire span of the musical piece. However, the piece lends itself to being divided on the time axis into bins/epochs (of varying widths), on the basis of music theory. An additional (and perhaps more meaningful) measure would thus also be to compute the inter-rater/subject agreement/reliability separately for each bin/epoch. Would the employed statistic (e.g. Kendall's W) change in that case?

  • $\begingroup$ Just wondering if my question is not clear enough, if I should give more detail - or on the contrary if there's too much detail and I should simplify the question? Slightly disappointing to see no replies even after creating a bounty.. :) $\endgroup$ – z8080 Apr 7 '16 at 12:11

A quick thought: perhaps consider @Jeffrey's answer if you have clear points you would like to pick, but perhaps a cluster analysis technique if you would like the data to decide. Here is a running mental dialogue as I read your problem (hopefully it helps):

People are pressing a button when they feel an emotion based on a stimulus in the song, but they aren't perfect. Thus, to account for the imperfection, we should allow a "range" where people are responding to the same stimulus but not responding at exactly the same time due to just pressing the button quicker or slower (incidentally, all button presses will, by definition, happen after the stimulus). How, then, do we define these clusters (where they are responding to the same stimulus)?

You could probably just eyeball them on the 1 dimensional scale (with all of the points plotted on one graph together), or you could use a statistical technique (enter the cluster analysis). I'd run it specifying small errors (and not the number of clusters), and do it until the results match the patterns that you think you're seeing (kindof assumes this is a pilot set, but, if you don't have any other info, probably the best you can do).

Finally, once you have the clusters (and their centers), you have a defined set of stimuli that the participants are responding to. This should allow traditional methods of inter-rater reliability, where each participant either responds to the "stimulus" or does not.

Again, a bit hacked together, but it lets your data guide where each "stimulus" is based off of the participant response.

  • $\begingroup$ A tolerance time-window (that extends only positively i.e. to the right), to account for subjects' delayed reactions is indeed appropriate. This would give me a slightly fairer picture of what I have now, which is, as @Jeffrey suggests, a binary time series across all bins, for each subject. $\endgroup$ – z8080 Apr 11 '16 at 8:33
  • $\begingroup$ What I came up with is to simply count, for each of the events of interest (hypothesised by my musical model), how many subjects press the key within the corresponding bin, and see which event has the best "popular support". This idea is admittedly simplistic, but would traditional indices of inter-rater reliability (accuracy, kappa, pi, S) really give a better measure? Again, what I want to quantify is for which of the bins the model correctly predicts that many subjects will press a key there. $\endgroup$ – z8080 Apr 11 '16 at 8:35
  • $\begingroup$ @wildetudor These are different questions. Agreement indices will tell you how much, on average, raters tended to press or not press the button during the same bins. They will not tell you which bin had the most or the fewest button presses. $\endgroup$ – Jeffrey Girard Apr 11 '16 at 20:30
  • $\begingroup$ @wildetudor I agree with Jeffrey: these measure two different things. Your initial question stated "quantify the amount of agreement that exists between subjects". This would be inter-rater reliability. Your problem's primary complexity, from my perspective, is solely due to determining characteristics for the raters to agree on, which either bins or clusters solve. Perhaps, for details on the differences between the proportion and IRR, it would be helpful to consider how these are calculated. $\endgroup$ – mflo-ByeSE Apr 11 '16 at 23:47

I recommend dividing each time-series into temporal bins. Each bin would take on a value of 1 if the rater pressed the button during that bin and 0 otherwise. You now have a comparable time-series for each rater and can use a traditional index of inter-rater reliability such as accuracy, kappa, pi, or S. Another option would be to calculate the total number of button presses per rater (across the whole tasks or segments of it) and compare these counts using an intraclass correlation coefficient (ICC). This answers a different question than the previous solution, but also a potentially interesting one.

See my website mReliability for more information and code for computing these indices in MATLAB. If you prefer R, you can try the irr package.

  • $\begingroup$ many thanks, please see my replies below to mfloren's answer as otherwise we are having two separate discussions :) $\endgroup$ – z8080 Apr 11 '16 at 8:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.