Is median fairer than mean?

Question

I recently read the advice that you should generally use median not mean to eliminate outliers. Example: The following article http://www.amazon.com/Forensic-Science-Introduction-Scientific-Investigative/product-reviews/1420064932/

has 16 reviews at the moment:

review = c(5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 3, 2, 1, 1)
summary(review)  ## "ordinary" summary

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
1.000   3.750   5.000   4.062   5.000   5.000 

Because they use Mean the article gets 4 stars but if they would use Median it would get 5 stars.

Isn't the median a 'fairer' judge?

---

An experiment shows that median's error is always bigger than mean. Is median worse?

library(foreach)

# the overall population of bookjudgments
n <- 5
p <- 0.5
expected.value <- n*p
peoplesbelieve <- rbinom(10^6,n, p)

# 16 ratings made for 100 books
ratings <- foreach(i=1:100, .combine=rbind) %do% sample(peoplesbelieve,16)
stat    <- foreach(i=1:100, .combine=rbind) %do% c(mean=mean(ratings[i,]), 
                   median=median(ratings[i,]))

# which mean square error is bigger? Mean's or Median's?
meansqrterror.mean   <- mean((stat[,"mean"]-expected.value)^2)
meansqrterror.median <- mean((stat[,"median"]-expected.value)^2)

res <- paste("mean MSE",meansqrterror.mean)
res <- paste(res, "| median MSE", meansqrterror.median)
print(res)

Answer 1

The problem is that you haven't really defined what it means to have a good or fair rating. You suggest in a comment on @Kevin's answer that you don't like it if one bad review takes down an item. But comparing two items where one has a "perfect record" and the other has one bad review, maybe that difference should be reflected.

There's a whole (high-dimensional) continuum between median and mean. You can order the votes by value, then take a weighted average with the weights depending on the position in that order. The mean corresponds to all weights being equal, the median corresponds to only one or two entries in the middle getting nonzero weight, a trimmed average corresponds to giving all except the first and last couple the same weight, but you could also decide to weight the $k$th out of $n$ samples with weight $\frac{1}{1 + (2 k - 1 - n)^2}$ or $\exp(-\frac{(2k - 1 - n)^2}{n^2})$, to throw something random in there. Maybe such a weighted average where the outliers get less weight, but still a nonzero amount, could combine good properties of median and mean?

Answer 2

The answer you get depends on the question you ask.

Mean and median answer different questions. So they give different answers. It's not that one is "fairer" than another. Medians are often used with highly skewed data (such as income). But, even there, sometimes the mean is best. And sometimes you don't want ANY measure of central tendency.

In addition, whenever you give a measure of central tendency, you should give some measure of spread. The most common pairings are mean-standard deviation and median-interquartile range. In these data, giving just a median of 5 is, I think, misleading, or, at least, uninformative. The median would also be 5 if every single vote was a 5.

Answer 3

If the only choices are integers in the range 1 to 5, can any really be considered an outlier?

I'm sure that with small sample sizes, popular outlier tests will fail, but that just points out the problems inherent in small samples. Indeed, given a sample of 5, 5, 5, 5, 5, 1, Grubbs' test reports 1 as an outlier at $\alpha = 0.05$. The same test for the data you give above does not identify the 1's as outliers.

Grubbs test for one outlier

data:  review  G = 2.0667, U = 0.6963,
p-value = 0.2153 alternative
hypothesis: lowest value 1 is an outlier

Answer 4

An experiment shows that median's error is always bigger than mean.

It depends on cost function you use.

MSE is minimized by mean. Therefore if you use MSE median will be always worse than mean.

BUT, if you would use absolute error, than the mean would be worse!

A nice explanation on this can be found here: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medians-and-means-an-unifying-perspective/

The choice depends on your problem and preferences. If you don't want outliers to have big impact on the position of the "central point", then you choose median. If you care about outliers, you choose mean.

Answer 5

Just a quick thought:

If you assume that each rating is drawn from a latent continuous variable, then you could define the median of this underlying continuous variable of interest as your value of interest, rather than the mean of this underlying distribution. Where the distribution is symmetric, then the mean and the median would ultimately be estimating the same quantities. Where the distribution is skewed, the median would differ from the mean. In this case, to my mind, the median would correspond more with what we think of as the typical value. This goes some way to understanding why median income and median house prices are typically reported rather than the mean.

However, when you have a small number of discrete values, the median performs poorly.

Perhaps, you could use some density estimation procedure and then take the median of that, or use some interpolated median.

Answer 6

The great thing about using the median for star ratings is that smart users (aware of the use of the median) won't "game" the system:

If a rational user thinks the proper rating should be 4 stars, but it currently has 4.5 stars, then the best way to get to four stars (assuming there have been more than six votes) is to vote 1 star in a mean based rating system.

While in a median based system, the user's rational choice is simply to vote exactly the number of stars the user thinks the product should have.

It's kind of the second price auction equivalent for star rating systems.

Answer 7

Several good answers still leave room for more comments.

First, no one has objected to the idea that the median is intended to eliminate outliers, but I will qualify it. The intended meaning is evident, but it is easy for real data to be more complicated. At most, the median is intended to discount or ignore outliers, but even that is not guaranteed. For example, with ratings of 1 1 1 5 5 5 the median and mean agree at 3, so all may seem good. But an extra 5 will tip the median to 5 and an extra 1 will tip the median to 1. The mean would move by about 0.286 in each case. Hence the mean is here more resistant than the median. The example can be dismissed as unusual, but it's not outrageous. The point is not original, naturally. One place it is made is in Mosteller, F. and Tukey, J.W. 1977. Data Analysis and Regression. Reading, MA: Addison-Wesley, pp.34-35.

Second, trimmed means have been mentioned and the idea deserves a bigger push. Mean and median need not be stark alternatives so that the analyst must choose (vote for) one or the other. You can consider all possible trimmed means based on trimming a certain number of values in each tail. The table shows as # the number of values included in the calculation of the mean:

  +----------------------------+
  | number    #   trimmed mean |
  |----------------------------|
  |      0   16         4.0625 |
  |      1   14       4.214286 |
  |      2   12       4.416667 |
  |      3   10            4.6 |
  |      4    8           4.75 |
  |      5    6       4.833333 |
  |      6    4              5 |
  |      7    2              5 |
  +----------------------------+

The main picture here is that you can choose your discount rate (ignore so many values in each tail as suspect) as a kind of insurance against the risk of being off because of extreme values. What I see is a fairly smooth gradient between mean and median, which is expected here because the possible values 1, 2, 3, 4, 5 are all present in the data. A big jump in the sequence is expected with an isolated outlier.

There is no obligation with trimmed means to trim equal numbers in each tail, but I will not expand on that.

Third, the example is of Amazon reviews. Context is always pertinent in guiding how you want data summarized. In the case of Amazon reviews the best answer is to read the reviews! As high and low grades alike can be on spurious grounds (implicitly: the author of this book is my friend) and/or irrelevant to your decision (explicitly: the re-seller treated me badly), there isn't to me an obvious implication for how to summarize such data, and indeed by showing you the distribution Amazon is being maximally informative.

Fourth, and most elementary but also fundamental of all, who is making you choose? Sometimes mean and median should both be reported (and, as said, a distribution graph too).