2
$\begingroup$

I'm trying to compute a reliability rating for potentially malicious emails. Essentially a system where multiple people will view a suspect email, and assign it a "malice score". The idea is for emails where the raters reliably rate it malicious, to flag it and do stuff to it (discard, mark for review etc.)

I've (tried to) read up about the various options...Kippendorf's Alpha, ICC etc, but these all need the same raters to rate multiple samples (which will not occur in my scenario - there will just be lots of samples being rated by lots of randomly selected users).

Can anyone suggest something to try? (And/or let me know if I've completely misunderstood!)

On consideration, I think I need to elaborate more: The goal is to quantify the degree of consensus among the random sample of raters for each email. With that information, we can automate an action for each email: e.g. If there is consensus the the email is bad/good, discard/allow. If there is significant disagreement, quarantine.

Any interest in whether or not the whole system of randomly selected raters + randomly selected emails is overall reliable is more of a side-topic.

Does that change anything?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ If I'm reading your second paragraph correctly: a random sample of raters will assess a random sample of emails, where raters do not rate the same set of emails (conversely, emails are only rated by a subsample of the pool of raters). Is that right? If so, things simplify a lot if this fall under a balanced incomplete block design. $\endgroup$
    – chl
    Commented Nov 22, 2012 at 10:32
  • 1
    $\begingroup$ If I've understood correctly, you can calculate the ICC using a mixed model framework (e.g. lmer). $\endgroup$
    – Jeremy Miles
    Commented Sep 7, 2017 at 16:19

1 Answer 1

Reset to default
2
$\begingroup$

I have the same issue where I want to compute per-item reliability, and as you said it looks like e.g., Krippendorf's alpha is not workable (as far as I can tell the expected disagreement is estimated across all units, so with only one unit the expected disagreement cancels the observed disagreement exactly).

My plan is to just fall back on raw percent agreement, averaged across all possible pairs of raters (see this article for a good example and breakdown of the following methods: percent agreement, Scott's Pi and Cohens Kappa, Fleiss' Kappa, and Krippendorf's Alpha).

I think that for your specific application simple percent agreement would work fine.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ It would be good to focus this answer more on the original question. Including a summary beyond the link would also help readers determine whether it includes methods they have not already tried. $\endgroup$
    – combo
    Commented Sep 7, 2017 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.