2
$\begingroup$

I'm trying to compute a reliability rating for potentially malicious emails. Essentially a system where multiple people will view a suspect email, and assign it a "malice score". The idea is for emails where the raters reliably rate it malicious, to flag it and do stuff to it (discard, mark for review etc.)

I've (tried to) read up about the various options...Kippendorf's Alpha, ICC etc, but these all need the same raters to rate multiple samples (which will not occur in my scenario - there will just be lots of samples being rated by lots of randomly selected users).

Can anyone suggest something to try? (And/or let me know if I've completely misunderstood!)

On consideration, I think I need to elaborate more: The goal is to quantify the degree of consensus among the random sample of raters for each email. With that information, we can automate an action for each email: e.g. If there is consensus the the email is bad/good, discard/allow. If there is significant disagreement, quarantine.

Any interest in whether or not the whole system of randomly selected raters + randomly selected emails is overall reliable is more of a side-topic.

Does that change anything?

$\endgroup$
2
  • 1
    $\begingroup$ If I'm reading your second paragraph correctly: a random sample of raters will assess a random sample of emails, where raters do not rate the same set of emails (conversely, emails are only rated by a subsample of the pool of raters). Is that right? If so, things simplify a lot if this fall under a balanced incomplete block design. $\endgroup$
    – chl
    Nov 22 '12 at 10:32
  • $\begingroup$ If I've understood correctly, you can calculate the ICC using a mixed model framework (e.g. lmer). $\endgroup$ Sep 7 '17 at 16:19
1
$\begingroup$

I have the same issue where I want to compute per-item reliability, and as you said it looks like e.g., Krippendorf's alpha is not workable (as far as I can tell the expected disagreement is estimated across all units, so with only one unit the expected disagreement cancels the observed disagreement exactly).

My plan is to just fall back on raw percent agreement, averaged across all possible pairs of raters (see this article for a good example and breakdown of the following methods: percent agreement, Scott's Pi and Cohens Kappa, Fleiss' Kappa, and Krippendorf's Alpha).

I think that for your specific application simple percent agreement would work fine.

$\endgroup$
1
  • 1
    $\begingroup$ It would be good to focus this answer more on the original question. Including a summary beyond the link would also help readers determine whether it includes methods they have not already tried. $\endgroup$
    – combo
    Sep 7 '17 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.