Check outliers in a set of proportions Suppose I have 100 people each of whom are given 100 randomly sampled emails to decide whether an email is a spam or not (each person gets a different set of 100 emails, so each email will be looked at by only one person). Each person then will have a proportion -- the proportion of emails labeled as spams. 
I wonder if there is a way to check whether some people's proportions are extremely different from the others'. Note that I am not concerned about the accuracy of each person. Rather, I am concerned about whether some people's decision patterns are very different from the bulk of the decision patterns. 
Using mean/standard deviation with proportions directly is probably not appropriate given that proportions have the lower and upper bounds of 0 and 1. I wonder if I can first transform proportions to logits and then use mean/standard deviation (say, anything outside mean_logit +/- 2*SD_logit).
 A: If all the people are consistent and there are $k$ spams in all $100\times 100 = 10^4$ e-mails, then the number of spams found in any randomly selected subset of $100$ of them would follow a Binomial distribution with parameters $n=100$ and $p=k/10^4$.  If someone reports a number $x$ that is out in the tails of this distribution, then it is unusually large or small.  (You can decide how far out in the tails to go to define "unusually.")
There are some complications, easily handled:


*

*You don't know the people are consistent.  Just estimate $k$ as the total number of spams they identify in toto.

*The $100$ samples you do have, although each is random, are not entirely independent (because they are mutually exclusive).  However, $100$ is so large you may ignore this.  You would start to be concerned with $10$ or fewer people.

*A value that might look unusual for a single person may arise at random when you examine a sufficiently large number of people.  This is the "multiple testing" problem.  The Bonferroni adjustment works well in this situation: if you want there to be only an $\alpha$ chance of mistaking a randomly large or small spam count as an outlier, then for one person you would set the thresholds at the $\alpha/2$ and $1-\alpha/2$ quantiles.  For instance, with $\alpha=0.05$ you would set them at the $2.5$ and $97.5$ percentiles.  Instead, divide $\alpha$ by the number of people.  With $100$ people you would use the $0.025$ and $99.975$ quantiles.
What does this prescription really amount to?  Let's explore the possibilities by looking at a wide range of possible values of the spam proportion $p$.  This figure shows the Binomial$(100,p)$ distributions for $p=0.05, 0.15, 0.30, 0.50, 0.80,$ and $0.90$.  (The distribution for $1-p$ is the reflection of that for $p$, so you get $6$ more distributions from this picture for free if you examine it in a mirror: $p=0.95, 0.85, 0.70, 0.20,$ and $0.10$.)

Each distribution assigns a positive probability to every whole number from $0$ through $100$, although many of those probabilities are extremely small.  The $\alpha=0.05$ outliers have been outlined in black.  The outlines are all down at a probability of $0$ because the corresponding values of $x$ are indeed all rare, as intended.
For example, if $k=8000$ spams are identified in the $10^4$ e-mails, estimate $p$ as $8000/10^4=0.80$.  The corresponding Binomial distribution is shown in blue-gray in the bottom middle panel.  The outliers are any values of $64$ or less (too small) or $93$ or greater (too large).  The chance of observing any such outlier in a random sample, when people are consistent, is just about one-twentieth of one percent.  But the chance of observing such an outlier in $100$ independent random samples is close to $\alpha=5\%$ as intended.  Thus, anyone who identifies such an extreme amount of spam may reasonably be suspected of evaluating emails differently than others: the randomness of the sample itself is not a plausible explanation for the difference.
Finally, if you identify many "outliers," take that as evidence that people are inconsistent. At this point it might no longer be useful to label any individual as an "outlier."  Instead, proceed to characterize the distribution of their results and see what you might learn from it.
A: As Yogi Berra supposedly said "You can see a lot by looking".  That is, look at your data. How high is the highest percentage? How high is the next highest? Are three gaps?  Do one or a few points stand out?  Those are outliers.
Sure, you can come up with various tests, but the best test is your brain and your eyes. 
However, if you want a formal test then you need a formal hypothesis to test.  So, you'd have to be precise about what you mean by "outlier".  That's not as easy as it sounds and your eyes will still be best. 
