Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to compute a reliability rating for potentially malicious emails. Essentially a system where multiple people will view a suspect email, and assign it a "malice score". The idea is for emails where the raters reliably rate it malicious, to flag it and do stuff to it (discard, mark for review etc.)

I've (tried to) read up about the various options...Kippendorf's Alpha, ICC etc, but these all need the same raters to rate multiple samples (which will not occur in my scenario - there will just be lots of samples being rated by lots of randomly selected users).

Can anyone suggest something to try? (And/or let me know if I've completely misunderstood!)

On consideration, I think I need to elaborate more: The goal is to quantify the degree of consensus among the random sample of raters for each email. With that information, we can automate an action for each email: e.g. If there is consensus the the email is bad/good, discard/allow. If there is significant disagreement, quarantine.

Any interest in whether or not the whole system of randomly selected raters + randomly selected emails is overall reliable is more of a side-topic.

Does that change anything?

share|improve this question
If I'm reading your second paragraph correctly: a random sample of raters will assess a random sample of emails, where raters do not rate the same set of emails (conversely, emails are only rated by a subsample of the pool of raters). Is that right? If so, things simplify a lot if this fall under a balanced incomplete block design. – chl Nov 22 '12 at 10:32
Exactly right. Thank you, I'll read up on that! – scuzzy-delta Nov 22 '12 at 20:37

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.