2
$\begingroup$

I am having some trouble trying to decide what measure of inter-rater reliability to use in a study. Part of a larger study involves accurately determining when participants began (onset) and stopped (offset) writing. I have video footage which captures the writing, and can determine the time of onset/offset to the nearest 0.01 second. The time is taken from a stopwatch which was running continuously from the start of each experiment, with multiple onset/offsets in each experiment. The onset/offset have been defined, and I would like to determine the inter-rater reliability for 2 raters. The data looks like this:

Participant A  trial one - onset: 21.35 offset: 24.55
               trial two - onset: 54.50 offset 57.75
Participant B  trial one - onset 10.35 offset 21.15
               trial two - onset 35.65 offset 39.90

and so on and so forth. Two raters would have data like the above and I would like to calculate if they are reliably similar. I looked at Kappa statistic, Krippendorff’s alpha, intra-class correlation, etc., but none seems appropriate for continuous time measurements.

$\endgroup$
2
$\begingroup$

It seems to me that what you're interested in is simply a measure of how much Rater 1's time measurements differ from Rater 2's, in which case you probably want to look at something simple like the root mean square difference in timings between the two raters. This will have the advantage of having an intuitive scale, being a measure (in seconds) of the typical difference between the two raters' timings.

$\endgroup$
  • $\begingroup$ Yes, that sounds almost exactly what I'm after.. but I need the added benefit of a statistic which will say whether the two raters are, statistically, acceptably similar to each other. Do you know what I mean? If I were to publish in a journal I need to be able to show that our timings are close enough to be considered real, and that the gods of statistics agree :) $\endgroup$ – Eric Mar 20 '12 at 20:51
  • 1
    $\begingroup$ Whether the raters are acceptably similar is a domain-specific matter: what might be considered acceptable in one field might not be in another. There's no objective standard of similarity here that can be used to make a judgment against. $\endgroup$ – Martin O'Leary Mar 20 '12 at 22:36
0
$\begingroup$

Perhaps you could use a hierarchical linear model with elapsed time nested within individual and individual cross nested within rater (assuming at least some individuals are evaluated by more than one rater). This would give you the variance at each level - with the variance at the rater level being the indicator of reliability/consistency.

I'm not positive on the exact structure of the data so forgive me if I'm missing something here.

Are individuals evaluated by more than one rater? Are you interested in the elapsed time or specifically the onset and the offset (as if they are independent of each other)?

$\endgroup$
  • $\begingroup$ The idea is that we randomly sample the data to ensure that the timings are correct. I'm not sure yet how many will be double timed, but the statistic will be carried out on only those that are. The key thing I'm after is having a statistic which will tell me that the two ratings are sufficiently similar enough to be considered reliable as a measurement. It's not elapsed time we're interested in but the onset and offset as independent measurements. $\endgroup$ – Eric Mar 20 '12 at 20:55
  • 1
    $\begingroup$ @Eric This isn't my area of expertise so caveat emptor. But I would go about this by first asking what level of precision is necessary. I would look at pairs of evaluations and calculate the mean and median divergence between raters (in seconds). If it's small then that tells you in intuitive units (seconds or fractions of a second) how far apart your raters are. Then it's a subjective call relative to your research question. $\endgroup$ – Will Mar 20 '12 at 21:37
  • $\begingroup$ I would consider truncation too. If your raters diverge, on average, by .3 seconds then consider truncating to the nearest second. For example, if rater 1 says 24.6 seconds and rater 2 says 24.3 seconds then they both become 24 seconds and there is no rater disagreement (on average). Of course you're sacrificing precision for inter-rater reliability. You have to make the call as to if this is an acceptable trade off. $\endgroup$ – Will Mar 20 '12 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.