Outlier management

Question

I apologize in advance for my novice question. I am a part of an interview committee of eight people. We interview 70 applicants for just six positions. All of the applicants are very accomplished. We rank order our applicants from best to worst. We have mostly just averaged our rankings to come up with a final rank list. We note that occasionally one interviewer will have a much different assessment of the applicant and their rank will be far from the others. Is there a proper way to deal with outliers in this setting? emphasized text

I see the method of calculating an IQR and deleting data outside that (mean - 1.5IQR to mean +1.5IQR) range makes sense in some situation but maybe not here. Each rating has value, even theses outliers. We don't want one opinion to overly bias the final results though. Would it make sense to change outliers to the lower or higher edge of the non-outlier zone? Are there better ways?

Answer 1

It's a good question. If you don't want to just keep that rating in the average (because it pulls it too hard on the average), but if you also don't want to completely delete it (therefore just tossing out any "dissenting" opinion), I would recommend trying to calculate the median rating, rather than the mean.

This way, their rating will still have an impact, but nowhere near the same heavy "pull" as when calculating mean.

EDIT: Curious about whuber's suggestion in the comments to this answer, I did a quick "back-of-the-envelope" test of median vs. Winsorized mean. I gave my code below. But essentially what I did is generate a normally distributed cluster of 7 ratings, and then added an outlier (randomly placed, but far away from the main cluster of ratings) on the 0-10 scale.

I then calculated the arithmetic mean, the median and the Winsorized mean, to see where the Wind. mean and the median end up relative to the arithmetic mean. I repeated this 1000 times just to get an average difference between the mean vs. Wind. Mean and median.

Just to give some examples

When the main group averaged a rating of 4 and the outlier was around 8, here's the average result: Arithmetic mean: 4.49 Median: 4.17 Wind. Mean: 4.24

When the main group averaged a rating of 9 and the outlier was around 3, here's the average result: Arithmetic mean: 8.17 Median: 8.83 Wind. Mean: 8.72

When the main group averaged a rating of 5 and the outlier was around 10, here's the average result: Arithmetic mean: 5.57 Median: 5.17 Wind. Mean: 5.23

So you can see that the median consistently gives slightly less weight to the outlier than Wind. mean, but generally the two results are quite close. Maybe it's splitting hairs here, I'm not sure.

Something to note, though, is that the median ratings also had a slighter higher variance (over the 1000 trials), than either the arithmetic mean or the Wind. mean.

Here's my R code:

    set.seed(100)
    
    arithmean <- c()
    windmean <- c()
    medians <- c()
    for (i in 1:1000){
      # Main rater cluster
      ratings <- rnorm(n=7, mean=3, sd=1)
      
      # Single outlier
      ratings <- c(ratings, rnorm(n=1, mean=8, sd=1))
      
      # Not allowing any ratings over 10 or under 0
      for (i in 1:length(ratings)){
        if (ratings[i] < 0){
          ratings[i] <- 0
        }else if (ratings[i] > 10){
          ratings[i] <- 10
        }
      }
      
      # Calculating the Winsorized Mean 
      sorted <- sort(ratings, decreasing=FALSE)
      sorted[1] <- sorted[2]
      sorted[8] <- sorted[7]
      windmean <- c(windmean, mean(sorted))
      
      # Calculating median
      medians <- c(medians, median(ratings))
      
      # Arithmetic mean
      arithmean <- c(arithmean, mean(ratings))
      
    }
    mean(arithmean)
    mean(medians)
    mean(windmean)
    
    sd(arithmean)
    sd(medians)
    sd(windmean)

EDIT 2:

I think whuber brings up an excellent point in the comments that in an ideal scenario, the raters' opinion should be weighted by some metric of how "valuable" it is. Perhaps it's worth trying to do a weighted average, where the weight is determined by some measure of the rater's experience/reliability, so that if a rater is an outlier but has great experience and insight, they might be weighted more heavily and vice versa. Maybe even a very simple implementation of this would be useful (ex: "senior" raters get 1.25x or 1.5x weight to their ratings).