I'm trying to compute a reliability rating for potentially malicious emails. Essentially a system where multiple people will view a suspect email, and assign it a "malice score". The idea is for emails where the raters reliably rate it malicious, to flag it and do stuff to it (discard, mark for review etc.)
I've (tried to) read up about the various options...Kippendorf's Alpha, ICC etc, but these all need the same raters to rate multiple samples (which will not occur in my scenario - there will just be lots of samples being rated by lots of randomly selected users).
Can anyone suggest something to try? (And/or let me know if I've completely misunderstood!)
On consideration, I think I need to elaborate more: The goal is to quantify the degree of consensus among the random sample of raters for each email. With that information, we can automate an action for each email: e.g. If there is consensus the the email is bad/good, discard/allow. If there is significant disagreement, quarantine.
Any interest in whether or not the whole system of randomly selected raters + randomly selected emails is overall reliable is more of a side-topic.
Does that change anything?