Inter-subject agreement re timings of (unequal number of) events

Question

I am trying to figure out what would be the best statistic to use to quantify the amount of agreement that exists between subjects who were asked to press a button whenever they felt a certain emotion while listening to a short (2min) piece of music. Plotted as time series, the raw data looks something like this (one subject per line; each event(keypress) represented by a blue dot):

I initially thought of Kendall's tau, but I believe that is only suited to rank data, which is not what I have here. I guess Kendall's W or Cronbach's alpha would be appropriate, although I am not sure whether these are robust to my data having unequal numbers of total events that the different subjects define, e.g. in the screenshot, the first subject defined 3 events, the second subject only one event, etc.

I think correlation would be unsuitable here, as I want to obtain the agreement between all subjects as opposed to between pairs of subjects.

I am also unsure whether what I need to quantify is in fact inter-rater agreement vs inter-rater reliability (e.g. see Liao et al 2010).

What I've described so far aims to obtain a single number that expresses the overall agreement between subjects across the entire span of the musical piece. However, the piece lends itself to being divided on the time axis into bins/epochs (of varying widths), on the basis of music theory. An additional (and perhaps more meaningful) measure would thus also be to compute the inter-rater/subject agreement/reliability separately for each bin/epoch. Would the employed statistic (e.g. Kendall's W) change in that case?

Answer 1

A quick thought: perhaps consider @Jeffrey's answer if you have clear points you would like to pick, but perhaps a cluster analysis technique if you would like the data to decide. Here is a running mental dialogue as I read your problem (hopefully it helps):

People are pressing a button when they feel an emotion based on a stimulus in the song, but they aren't perfect. Thus, to account for the imperfection, we should allow a "range" where people are responding to the same stimulus but not responding at exactly the same time due to just pressing the button quicker or slower (incidentally, all button presses will, by definition, happen after the stimulus). How, then, do we define these clusters (where they are responding to the same stimulus)?

You could probably just eyeball them on the 1 dimensional scale (with all of the points plotted on one graph together), or you could use a statistical technique (enter the cluster analysis). I'd run it specifying small errors (and not the number of clusters), and do it until the results match the patterns that you think you're seeing (kindof assumes this is a pilot set, but, if you don't have any other info, probably the best you can do).

Finally, once you have the clusters (and their centers), you have a defined set of stimuli that the participants are responding to. This should allow traditional methods of inter-rater reliability, where each participant either responds to the "stimulus" or does not.

Again, a bit hacked together, but it lets your data guide where each "stimulus" is based off of the participant response.

Answer 2

I recommend dividing each time-series into temporal bins. Each bin would take on a value of 1 if the rater pressed the button during that bin and 0 otherwise. You now have a comparable time-series for each rater and can use a traditional index of inter-rater reliability such as accuracy, kappa, pi, or S. Another option would be to calculate the total number of button presses per rater (across the whole tasks or segments of it) and compare these counts using an intraclass correlation coefficient (ICC). This answers a different question than the previous solution, but also a potentially interesting one.

See my website mReliability for more information and code for computing these indices in MATLAB. If you prefer R, you can try the irr package.