Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I recently read the advice that you should generally use median not mean to eliminate outliers. Example: The following article http://www.amazon.com/Forensic-Science-Introduction-Scientific-Investigative/product-reviews/1420064932/

has 16 reviews at the moment:

review= c(5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 3, 2, 1, 1)
summary(review)  ## "ordinary" summary

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
1.000   3.750   5.000   4.062   5.000   5.000 

Because they use Mean the article gets 4 stars but if they would use Median it would get 5 stars.

Isn't the median a 'fairer' judge?


An experiment shows that median's error is always bigger than mean. Is median worse?

library(foreach)

#the overall population of bookjudgments
n<-5
p<-0.5
expected.value<-n*p
peoplesbelieve <-rbinom(10^6,n, p)

#16 ratings made for 100 books
ratings <- foreach(i=1:100, .combine=rbind) %do% sample(peoplesbelieve,16)
stat <- foreach(i=1:100, .combine=rbind) %do% c(mean=mean(ratings[i,]), median=median(ratings[i,]))

#which mean square error is bigger? Mean's or Median's?
meansqrterror.mean<-mean((stat[,"mean"]-expected.value)^2)
meansqrterror.median<-mean((stat[,"median"]-expected.value)^2)

res<-paste("mean MSE",meansqrterror.mean)
res<-paste(res, "| median MSE", meansqrterror.median)
print(res)

share|improve this question
2  
Why would having a 5 star rating be fairer? There are 6/16 reviews that gave a lower rating... –  nico Feb 5 '11 at 17:00
    
Ok, than you think Mean is the right average? the majority said its 5. 60% more the rest 6/16 said so. –  Roland Kofler Feb 5 '11 at 17:08
2  
If I had to give a discrete evaluation, seeing those 16 reviews I would give 4, not 5, as -to me- 5 would mean that all (or the great majority) of the votes are 5. 6/16 is ~40%, which is not exactly negligible. –  nico Feb 5 '11 at 18:22
4  
So, in essence, I think neither mean or median are good. Showing (as Amazon does) a bargraph with the different votes is the best option. Also, it is interesting to point out that online 1-5 ratings are not always so fair... youtube-global.blogspot.com/2009/09/… –  nico Feb 5 '11 at 18:27
1  
@nico: scoring is full of traps, what you point out is one of the arguments of my article here: objektorient.blogspot.com/2010/09/… –  Roland Kofler Feb 5 '11 at 19:56

5 Answers 5

up vote 21 down vote accepted

The problem is that you haven't really defined what it means to have a good or fair rating. You suggest in a comment on @Kevin's answer that you don't like it if one bad review takes down an item. But comparing two items where one has a "perfect record" and the other has one bad review, maybe that difference should be reflected.

There's a whole (high-dimensional) continuum between median and mean. You can order the votes by value, then take a weighted average with the weights depending on the position in that order. The mean corresponds to all weights being equal, the median corresponds to only one or two entries in the middle getting nonzero weight, a trimmed average corresponds to giving all except the first and last couple the same weight, but you could also decide to weight the $k$th out of $n$ samples with weight $\frac{1}{1 + (2 k - 1 - n)^2}$ or $\exp(-\frac{(2k - 1 - n)^2}{n^2})$, to throw something random in there. Maybe such a weighted average where the outliers get less weight, but still a nonzero amount, could combine good properties of median and mean?

share|improve this answer

The answer you get depends on the question you ask.

Mean and median answer different questions. So they give different answers. It's not that one is "fairer" than another. Medians are often used with highly skewed data (such as income). But, even there, sometimes the mean is best. And sometimes you don't want ANY measure of central tendency.

In addition, whenever you give a measure of central tendency, you should give some measure of spread. The most common pairings are mean-standard deviation and median-interquartile range. In these data, giving just a median of 5 is, I think, misleading, or, at least, uninformative. The median would also be 5 if every single vote was a 5.

share|improve this answer
3  
Your point about a measure of spread is a key here. That's one of the issues that keeps coming up in this discussion, under other names, and it also ties in with Erik P's discussion of weighting schemes. –  Wayne Feb 6 '11 at 22:47

If the only choices are integers in the range 1 to 5, can any really be considered an outlier?

I'm sure that with small sample sizes, popular outlier tests will fail, but that just points out the problems inherent in small samples. Indeed, given a sample of 5, 5, 5, 5, 5, 1, Grubbs' test reports 1 as an outlier at $\alpha = 0.05$. The same test for the data you give above does not identify the 1's as outliers.

Grubbs test for one outlier

data:  review  G = 2.0667, U = 0.6963,
p-value = 0.2153 alternative
hypothesis: lowest value 1 is an outlier
share|improve this answer
    
because just one person in your example can change the result dramatically. if the person believed the book has a different topic his fault will change the rating –  Roland Kofler Feb 5 '11 at 17:11
1  
Is someone's opinion a fault? I'd argue that the failure is trying to draw meaningful conclusions based on a single statistic from only a few data points. As noted by @nico above, Amazon does show a bar graph of all the ratings. –  kmm Feb 7 '11 at 16:14

Just a quick thought:

If you assume that each rating is drawn from a latent continuous variable, then you could define the median of this underlying continuous variable of interest as your value of interest, rather than the mean of this underlying distribution. Where the distribution is symmetric, then the mean and the median would ultimately be estimating the same quantities. Where the distribution is skewed, the median would differ from the mean. In this case, to my mind, the median would correspond more with what we think of as the typical value. This goes some way to understanding why median income and median house prices are typically reported rather than the mean.

However, when you have a small number of discrete values, the median performs poorly.

Perhaps, you could use some density estimation procedure and then take the median of that, or use some interpolated median.

share|improve this answer

An experiment shows that median's error is always bigger than mean.

It depends on cost function you use.

MSE is minimalized by mean. Therefore if you use MSE median will be always worse than mean.

BUT, if you would use absolute error, than the mean would be worse!

A nice explanation on this can be found here: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medians-and-means-an-unifying-perspective/

The choice depends on your problem and preferences. If you don't want outliers to have big impact on the position of the "central point", then you choose median. If you care about outliers, you choose mean.

share|improve this answer
    
+1 Good points. Welcome to our site, Dominik! –  whuber Jan 5 at 16:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.