50
$\begingroup$

If I understand correctly, book ratings on a 1-5 scale are Likert scores. That is, a 3 for me may not necessarily be a 3 for someone else. It's an ordinal scale IMO. One shouldn't really average ordinal scales but can definitely take the mode, median and percentiles.

So is it 'okay' to bend the rules since the large part of the population understands means than the above statistics? Although the research community strongly rebukes taking averages of Likert scale based data, is it fine to do this with the masses (practically speaking)? Is taking the average in this case even misleading to start with?

Seems unlikely that a company like Amazon would fumble on basic statistics, but if not then what am I missing here? Can we claim that the ordinal scale is a convenient approximation to the ordinal to justify taking the mean? On what grounds?

Improve this question
$\endgroup$
7
  • 3
    $\begingroup$ If the 3 for you is not the same as a 3 for somebody else, then you don't even have a scale: you have a collection of incomparable measurements and there's little meaningful you can do to summarize them. What makes a scale ordinal is that (a) values can be compared, so your 3 and my 3 mean the same thing, but (b) numerical differences of values are meaningless apart from their signs, so that (say) two 3's, a 4 and a 2, or a 5 and a 1 could be put in any order, although numerically each pair of ratings has the same mean and median. $\endgroup$
    – whuber ♦
    Commented Jul 3, 2012 at 22:20
  • 1
    $\begingroup$ @whuber - but isn't it true that 2 people may not share the same opinion on a 1-9 scale about the numbers? A 6 for me may indeed not be a 6 for someone else unless they have a predefined scale to go by? $\endgroup$
    – PhD
    Commented Jul 3, 2012 at 22:35
  • 1
    $\begingroup$ I read one review recently on amazon that said "Brilliant product can't fault it. I would never give anything 5 stars, so have awarded 4". If this doesn't skew the mean then I don't know that does $\endgroup$
    – Matt Wilko
    Commented Jul 4, 2012 at 15:28
  • 2
    $\begingroup$ @Wilko You are talking about differences of opinion, not differences of scale. Even when a scale is very carefully calibrated, as in (say) scoring for gymnastics or figure skating or the international scale for rating the difficulty of rapids on rivers, and even when experts are trained to use that scale, there will still be variation. That is not usually interpreted as evidence the scale is subjective: it's interpreted as variation among the judges. $\endgroup$
    – whuber ♦
    Commented Jul 4, 2012 at 18:03
  • 1
    $\begingroup$ Sorry, this is not really an answer, but unfortunately I could not find the "comment"-function. Recently, I have started to write my master thesis about the key elements of customer reviews. In consideration of the following circumstances I also started to doubt the significance of Amazon's 5-star rating system. - Number of distrustful reviews - Effects of the ratings bias and J-Curves (buildingreputation.com/writings/2009 $\endgroup$
    – derPio
    Commented Jul 5, 2012 at 9:19

7 Answers 7

Reset to default
44
$\begingroup$

Benefits of using the mean to summarise central tendency of a 5 point rating

As @gung mentioned I think there are often very good reasons for taking the mean of a five-point item as an index of central tendency. I have already outlined these reasons here.

To paraphrase:

  1. the mean is easy to calculate
  2. The mean is intuitive and well understood
  3. The mean is a single number
  4. Other indices often yield similar rank ordering of objects

Why the mean is good for Amazon

Think about the goals of Amazon in reporting the mean. They might be aiming to

  • provide an intuitive and understandable rating for an item
  • ensure user acceptance of the rating system
  • ensure that people understand what the rating means so they can use it appropriately to inform purchasing decisions

Amazon provides some sort of rounded mean, frequency counts for each rating option, and the sample size (i.e., number of ratings). This information presumably is enough for most people to appreciate both the general sentiment regarding the item and the confidence in such a rating (i.e., a 4.5 with 20 ratings is more likely to be accurate than a 4.5 with 2 ratings; an item with 10 5-star ratings, and one 1-star rating with no comments might still be a good item).

You could even see the mean as a democratic option. Many elections are decided based on which candidate gets the highest mean on a two-point scale. Similarly, if you take the argument that each person who submits a review gets a vote, then you can see the mean as a form that weights each person's vote equally.

Are differences in scale use really a problem?

There are a wide range of rating biases known in the psychological literature (for a review, see Saal et al 1980), such as central tendency bias, leniency bias, strictness bias. Also, some raters will be more arbitrary and some will be more reliable. Some may even systematically lie giving fake positive or fake negative reviews. This will create various forms of error when trying to calculate the true mean rating for an item.

However, if you were to take a random sample of the population, such biases would cancel out, and with a sufficient sample size of raters, you would still get the true mean.

Of course, you don't get a random sample on Amazon, and there is the risk that the particular set of raters you get for an item is systematically biased to be more lenient or strict and so on. That said, I think users of Amazon would appreciate that user submitted ratings come from an imperfect sample. I also think that it's quite likely that with a reasonable sample size that in many cases, the majority of response bias differences would start to disappear.

Possible advances beyond the mean

In terms of improving the accuracy of the rating, I wouldn't challenge the general concept of the mean, but rather I think there are other ways of estimating the true population mean rating for an item (i.e., the mean rating that would be obtained were a large representative sample asked to rate the item).

  • Weight raters based on their trustworthiness
  • Use a Bayesian rating system that estimates the mean rating as a weighted sum of the average rating for all items and the mean from the specific item, and increase the weighting for the specific item as the number of ratings increases
  • Adjust the information of a rater based on any general rating tendency across items (e.g., a 5 from someone who typically gives 3s would be worth more than someone who typically gives 4s).

Thus, if accuracy in rating was the primary goal of Amazon, I think it should endeavour to increase the number of ratings per item and adopt some of the above strategies. Such approaches might be particularly relevant when creating "best-of" rankings. However, for the humble rating on the page, it may well be that the sample mean better meets the goals of simplicity and transparency.

References

  • Saal, F.E., Downey, R.G. & Lahey, M.A. (1980). Rating the ratings: Assessing the psychometric quality of rating data.. Psychological Bulletin, 88, 413.
Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ +1. I think this goes beyond / extends your previous answer in a very nice way. I especially like the section re 'why the mean is good for Amazon', which enumerates more clearly what I was trying to get at in my last sentence. 'Different uses of a scale' is also quite insightful; I would appreciate a cite to an review of that literature, if you know of a good one. I note though, that the last section is somewhat in tension w/ the 2nd. $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 4, 2012 at 3:47
  • 2
    $\begingroup$ Thanks. I added a reference to the rating bias literature, and added something at the end that tries to reconcile the two perspectives. $\endgroup$
    – Jeromy Anglim
    Commented Jul 4, 2012 at 4:00
  • 2
    $\begingroup$ +1 @JeromyAnglim - a thorough perspective throwing light on the various aspects of the problem. Kudos! $\endgroup$
    – PhD
    Commented Jul 4, 2012 at 4:16
  • $\begingroup$ +1, great answer. Though I did find one sentence slightly misleading. When you said "However, if you were to take a random sample of the population, such biases would cancel out, and with a sufficient sample size of raters, you would still get the true mean." - I don't think that applies to all biases even if you had a random sample of the population. $\endgroup$
    – Michael Bishop
    Commented Jul 4, 2012 at 21:01
  • 1
    $\begingroup$ @MichaelBishop Thanks, I agree my language was a bit sloppy there. I guess it depends on what is meant by the "true mean". I can see how if you have fakers in the population this could bias the unadjusted population mean away from a hypothetical "true mean". I was thinking more that any systematic biases of individuals that apply to all items would cancel out to enable unbiased rank ordering of items based on the resulting mean. $\endgroup$
    – Jeromy Anglim
    Commented Jul 4, 2012 at 23:55
16
$\begingroup$

Everyone has good opinions on this. I don't really think I can add very much more. However, I will post this:

Improve this answer
$\endgroup$
4
  • 7
    $\begingroup$ I guess the comic highlights that some people are poor judges of the quality of an item, and by averaging over many such people, you get a poor average. In general wisdom of crowds suggests that averages perform fairly well where at least a reasonable proportion of people have some knowledge. Weighting ratings by trustworthiness might also be one strategy for overcoming issues. $\endgroup$
    – Jeromy Anglim
    Commented Jul 4, 2012 at 4:10
  • 1
    $\begingroup$ The other option is using Netflix style recommendations, by comparing your rating with the ratings of other users, and then averaging ratings offered by users with similar choices as you. $\endgroup$
    – rahul
    Commented Jul 4, 2012 at 5:03
  • 1
    $\begingroup$ @rahul That's a good point. In my answer I at times assume that ratings are largely true score + error, even if there is structure to the error. But when it comes to domains where personal preference is part of the definition of quality, this doesn't always make as much sense. $\endgroup$
    – Jeromy Anglim
    Commented Jul 4, 2012 at 5:49
  • $\begingroup$ I like that, and it is why (as a consumer) I try to read the reviews and not just look at the number of stars. But I thought it was ironic that in this case the more "sophisticated" methods of median, mode and percentiles all give a worse result than mean ;-) $\endgroup$
    – Darren Cook
    Commented Jul 6, 2012 at 0:03
15
$\begingroup$

To be somewhat technical here, those ratings aren't actually a Likert scale; they're just ordinal ratings. Now, having said that, your point is essentially correct. However, I often think too much is made of this issue. One thing to note is that it is typically understood that the average of a number of ordinal items can be approximately interval, and thus, when there are many ratings the mean becomes a more reasonable representation. I have found this answer by @JeromyAnglim to be excellent (really, the question and all attendant answers there are worth reading). For a more theoretical treatment, see here. On a different note, I like Amazon, but I see no reason to expect statistical sophistication from them, especially in terms of basic site design--the point is usability by consumers, not to impress stats professors.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Amazon has been one of the leaders in the (internet) technology industry in experimental design for online advertising and website usage. You can be sure that they actually are quite sophisticated in their statistical approaches. :-) Your point is a good one. To take it a small step farther, can you imagine if Amazon were doing something "more sophisticated" and someone checked them by using a simple average, found that some items were ranked "higher" than their average and others "lower", raising a fuss and leaving Amazon to try to explain their "hidden biases" regarding products? $\endgroup$
    – cardinal
    Commented Jul 3, 2012 at 23:19
  • 1
    $\begingroup$ Other services, e.g., Netflix, avoid this problem by only providing the "summary" data. :) $\endgroup$
    – cardinal
    Commented Jul 3, 2012 at 23:22
  • $\begingroup$ @cardinal, that's very interesting, I didn't know that about Amazon. $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 4, 2012 at 1:29
3
$\begingroup$

In my experience, the mean of rating-scale data is often the most closely correlated with the level of real-world metrics we try to associate with the rating scale. We have found a lot of linear relationships, and the average is therefore one of the better ways to summarize the data. That being said, as Jeromy pointed out, most ways of analyzing the central tendency of a rating scale will give similar results (rank orders, etc) most of the time.

Also, I suspect Amazon is probably not all that concerned with the scientific validity one way or the other. Amazon's goal, in the end, is to get people to shop more on Amazon.com, and the way reviews help achieve that will probably not vary with whatever one-number summary is used. Good products will be rewarded, really bad products punished, and nervous purchasers will have a chance to review pros and cons in more detail.

Improve this answer
$\endgroup$
2
$\begingroup$

Amazon ratings are misleading due to companies gaming the system. When customers are offered rebates and free merchandise in return for 5-star reviews, the "statistics" of what the ratings number is or means become moot.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Do you have any data on how often such things happen? $\endgroup$
    – Michael Bishop
    Commented Jul 10, 2012 at 21:23
1
$\begingroup$

You make a good point. Taking the mean of ordinal numbers is somewhat misleading. Any summary of several rankings would suffer from the fact that my subjective 3 may really equate to your 4. So combining different individual scores is probably the biggest problem. Interpreting the average of a 3 and a 4 as 3.5 is not nearly as egregious.

Improve this answer
$\endgroup$
1
$\begingroup$

In surveys the 5 star system is a weighted system and not 4 stars is worth 80% of 100%, nor should it be used in this manner. Most people understand 4&5 means a strong buy, 3 means nuetral and 1&2 are negative. Therefor the calculation should be 5 for 5, 4 for 4, zero for 3, -1 for 2 and -2 for 1 star. This is a common use in surveys. If Amazon wants to correct it they need to move to a ±5/-5 system.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.