# Outlier management

I apologize in advance for my novice question. I am a part of an interview committee of eight people. We interview 70 applicants for just six positions. All of the applicants are very accomplished. We rank order our applicants from best to worst. We have mostly just averaged our rankings to come up with a final rank list. We note that occasionally one interviewer will have a much different assessment of the applicant and their rank will be far from the others. Is there a proper way to deal with outliers in this setting? emphasized text

I see the method of calculating an IQR and deleting data outside that (mean - 1.5IQR to mean +1.5IQR) range makes sense in some situation but maybe not here. Each rating has value, even theses outliers. We don't want one opinion to overly bias the final results though. Would it make sense to change outliers to the lower or higher edge of the non-outlier zone? Are there better ways?

• You could do the same as Olympic judging and toss out the high and low scores and average the remaining ones. This will remove the outlier and not bias the results. Ideally the new mean should be closer to the median. This is also known as a trimmed mean. Jan 29, 2022 at 1:47

It's a good question. If you don't want to just keep that rating in the average (because it pulls it too hard on the average), but if you also don't want to completely delete it (therefore just tossing out any "dissenting" opinion), I would recommend trying to calculate the median rating, rather than the mean.

This way, their rating will still have an impact, but nowhere near the same heavy "pull" as when calculating mean.

EDIT: Curious about whuber's suggestion in the comments to this answer, I did a quick "back-of-the-envelope" test of median vs. Winsorized mean. I gave my code below. But essentially what I did is generate a normally distributed cluster of 7 ratings, and then added an outlier (randomly placed, but far away from the main cluster of ratings) on the 0-10 scale.

I then calculated the arithmetic mean, the median and the Winsorized mean, to see where the Wind. mean and the median end up relative to the arithmetic mean. I repeated this 1000 times just to get an average difference between the mean vs. Wind. Mean and median.

Just to give some examples

When the main group averaged a rating of 4 and the outlier was around 8, here's the average result: Arithmetic mean: 4.49 Median: 4.17 Wind. Mean: 4.24

When the main group averaged a rating of 9 and the outlier was around 3, here's the average result: Arithmetic mean: 8.17 Median: 8.83 Wind. Mean: 8.72

When the main group averaged a rating of 5 and the outlier was around 10, here's the average result: Arithmetic mean: 5.57 Median: 5.17 Wind. Mean: 5.23

So you can see that the median consistently gives slightly less weight to the outlier than Wind. mean, but generally the two results are quite close. Maybe it's splitting hairs here, I'm not sure.

Something to note, though, is that the median ratings also had a slighter higher variance (over the 1000 trials), than either the arithmetic mean or the Wind. mean.

Here's my R code:

    set.seed(100)

arithmean <- c()
windmean <- c()
medians <- c()
for (i in 1:1000){
# Main rater cluster
ratings <- rnorm(n=7, mean=3, sd=1)

# Single outlier
ratings <- c(ratings, rnorm(n=1, mean=8, sd=1))

# Not allowing any ratings over 10 or under 0
for (i in 1:length(ratings)){
if (ratings[i] < 0){
ratings[i] <- 0
}else if (ratings[i] > 10){
ratings[i] <- 10
}
}

# Calculating the Winsorized Mean
sorted <- sort(ratings, decreasing=FALSE)
sorted <- sorted
sorted <- sorted
windmean <- c(windmean, mean(sorted))

# Calculating median
medians <- c(medians, median(ratings))

# Arithmetic mean
arithmean <- c(arithmean, mean(ratings))

}
mean(arithmean)
mean(medians)
mean(windmean)

sd(arithmean)
sd(medians)
sd(windmean)


EDIT 2:

I think whuber brings up an excellent point in the comments that in an ideal scenario, the raters' opinion should be weighted by some metric of how "valuable" it is. Perhaps it's worth trying to do a weighted average, where the weight is determined by some measure of the rater's experience/reliability, so that if a rater is an outlier but has great experience and insight, they might be weighted more heavily and vice versa. Maybe even a very simple implementation of this would be useful (ex: "senior" raters get 1.25x or 1.5x weight to their ratings).

• Many ratings systems -- ice skating, for instance -- instead automatically trim the extremes or Winsorize them. Why, then, recommend the median?
– whuber
Jan 28, 2022 at 19:25
• @whuber Ease of interpretability and computation? I would also wager that it would have a very similar result as Winsorizing. But it's just my recommendation :) Jan 28, 2022 at 19:38
• I think like Winsorizing, using median has similar result of "deferring" the outlier's effect to the nearest, less extreme ratings. Jan 28, 2022 at 19:42
• The potential problem with the median is that it is so crude: it is relatively insensitive to strong opinions expressed by some of the raters. What is really needed here is a consideration of how the raters ought to be treated. In some circumstances, perhaps many, an organization will want to weight the raters according to their reliability and experience and it might want to allow some outlying (but not extremely outlying) ratings to stand.
– whuber
Jan 28, 2022 at 21:20