I have multiple independent coders who are trying to identify events in a time series -- in this case, watching video of face-to-face conversation and looking for particular nonverbal behaviors (e.g., head nods) and coding the time and category of each event. This data could reasonable be treated as a discrete-time series with a high sampling rate (30 frames/second) or as a continuous-time series, whichever is easier to work with.

I'd like to compute some measure of inter-rater reliability, but I expect there to be some uncertainty in when events occurred; that is, I expect that one coder might, for example, code that a particular movement began a quarter second later than other coders thought it started. These are rare events, if that helps; typically at least several seconds (hundreds of video frames) between events.

Is there a good way of assessing inter-rater reliability that looks at both of these kinds of agreement and disagreement: (1) do raters agree on what event occurred (if any), and (2) do they agree on when it occurred? The second is important to me because I'm interested in looking at the timing of these events relative to other things happening in the conversation, like what people are saying.

Standard practice in my field seems to be to divide things up into time slices, say 1/4 of a second or so, aggregate the events each coder reported per time slice, then compute Cohen's kappa or some similar measure. But the choice of slice duration is ad-hoc, and I don't get a good idea of uncertainty in time of events.

Best thought I have so far is that I could compute some kind of reliability curve; something like kappa as a function of the size of the window within which I consider two events as being coded at the same time. I'm not really sure where to go from there, though...

  • $\begingroup$ It looks like situation where functional data analysis methods can be applied. Did you consider them? $\endgroup$
    – mpiktas
    Dec 22, 2010 at 19:57
  • $\begingroup$ I had thought about functional data analysis, but it's not an area that I'm very familiar with. I'm working thru Ramsay and Silverman's book now. But I don't immediately see how to deal with a multinomial outcome variable...? $\endgroup$
    – dschulman
    Dec 23, 2010 at 16:58
  • $\begingroup$ Is there a gold standard available for those measurements (i.e., do you know when the event(s) of interest occur(s))? How many coders are included in this study? How many distinct events can we expect? $\endgroup$
    – chl
    Dec 23, 2010 at 21:37

2 Answers 2


Here's a couple of ways to think about.


A) You could treat each full sequence of codings as a ordered set of events (i.e. ["head nod", "head shake", "head nod", "eyebrow raised"] and ["head nod", "head shake", "eyebrow raised"]), then align the sequences using an algorithm that made sense to you ( http://en.wikipedia.org/wiki/Sequence_alignment ). You could then compute inter coder reliability for the entire sequence.

B) Then, again using the aligned sequences, you could compare when they said an event happened, given that they both observed the event.

2) Alternately, you could model this as a Hidden Markov Model, and use something like the Baumn-Welch algorithm to impute the probabilities that, given some actual event, each coder actually coded the data correctly. http://en.wikipedia.org/wiki/Baum-Welch_algorithm

  • $\begingroup$ This technique appears to be similar to what Grafsgaard 2012 did for similar sounding task. $\endgroup$
    – KevinL
    Mar 25, 2015 at 14:16

Rather than slicing the data up in arbitrary pieces you could consider the actual time differences. Coder 1 reports time and action:

049 D
113 C
513 C
724 G

A simple way to see which coder is the most reliable according to other coders is by giving him a score like so:

Add a point for each other coder that reported a D between (049-025) and (049+025)
Add a point for each other coder that reported a C between (113-025) and (113+025)
Add a point for each other coder that reported a C between (513-025) and (513+025)
Add a point for each other coder that reported a C between (724-025) and (724+025)
Subtract a point for each reported action.

If closeness is important for you, consider alternatives like these:

Add 25/(Time_Thiscoder-Time_Othercoder)^2 points for each other coder that reported a matching observation.

With all problem information available it should not be hard to implement this idea in a practical way.

  • 1
    $\begingroup$ Where do the "25"s come from? Do you have references or can you explain the theory that justifies this proposal? $\endgroup$
    – whuber
    Jan 2, 2012 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.