# How do I sort an ordinal list of user-generated ratings data?

The Ubuntu Software Center uses a 1 to 5 star rating system for its App Reviews. However, it's current ratings sorting algorithm looks very fishy. I believe this is a different question from this one, which seems to assume the mean of the ratings is a meaningful number.

If I assume that user supplied ratings are ordinal data, then doing things like adding the ratings together or taking their mean are not correct. The primary method of sorting must be the sample median.

Unfortunately, this leaves me with a lot of duplicates since 30 applications are being pigeon-holed into 5 stars of ratings, so it must be possible to further subsort the apps with identical median stars. I believe I want:

• Rating at the median should be better than rating below it.
• {2,3,3,3,4} > {2,2,3,3,4}
• Similarly, rating at the median should be worse than rating above it.
• {2,2,3,3,4} < {2,2,3,4,4}
• Ratings above and below the median should be equivalent.
• {1,1,3,4,4} = {2,2,3,4,4} = {2,2,3,5,5}
• Among two apps with the same median, higher confidence that the median is at least that large should rank higher.
• {2,3,3,3,4} > {2,3,4}

Are these reasonable desires? What algorithm can get me there?

My intuition tells me I want something like sample median + {lower bound probability estimate someone ranks that app higher than median} - {upper bound probability estimate someone ranks that app lower than median}. So:

• a large data set composed of 20 % 1's, 40% 3's, and 40% 4's would approach (3)+(2/5)-(1/5) = 3.2
• a set composed of equal parts 2,3, and 5 would approach (3)+(1/3)-(1/3) = 3.0
• and a set composed of 40% 1's and 60% 3's would approach (3)+(0)-(2/5) = 2.6.

Is this reasonable?

-
It appears this question is really a previous one in a different guise: stats.stackexchange.com/questions/9358/…. If that does not help, please feel free to edit your question to indicate how it is different. – whuber Nov 29 '11 at 18:35
I'm not sure I understand the comparison to that question as there's only one variable input here (a single rating from 1 to 5 stars). I only mean to imply that the data produced by that rating isn't linearly comparable (the difference between 1 and 2 stars is not the difference between 4 and 5 stars) – Scott Ritchie Nov 29 '11 at 20:39
On the contrary, your question is about vectors of ratings: there is one variable for each rating in your dataset. Achieving your goal will be tantamount to re-expressing the ratings in terms of numerical valuations and comparing each vector based on its mean valuation. Otherwise, you are going to come up with comparisons that have paradoxical properties; for example, they might not be transitive. – whuber Nov 29 '11 at 20:43
OK, let's compare medians. To resolve ties, use two natural principles: (a) dominance: when every value in set X, when ordered, is greater than or equal to corresponding values in Y (when ordered), X is preferred; and otherwise, (b) consistency: lower variances are preferred. (b) implies 2244 > 2335, (a) implies 2335 > 2334, and (b) implies 2334 > 2244 (all have a median of 3). This partial order is not transitive! What you're confronting here is a voting paradox. – whuber Nov 29 '11 at 21:51
The point, Scott, is that paradoxical orderings are possible, not that the one I pointed out is necessarily of interest to you. If you would like to inform your intuition and get a better sense of the challenges here (and the possible solutions) then consult the (extensive) literature on voting paradoxes and voting methods. But please don't expect ad hoc formulas necessarily to behave well, or you might be unhappily surprised. – whuber Nov 30 '11 at 2:39
show 4 more comments

## 2 Answers

You asked if your requirements look reasonable. My opinion: No, your requirements are not reasonable. In particular, you proposed:

Ratings above and below the median should be equivalent.

• {1,1,3,4,4} = {2,2,3,4,4} = {2,2,3,5,5}

This is not reasonable. {1,1,3,4,4} is a significantly worse score than {2,2,3,5,5}.

I have studied app market ratings, and found that there is a significant difference between what a 4 vs a 5 (in terms of what this tends to mean to the reviewer). For instance, a common meaning of 5 is "I love it"; a common meaning of 4 is "I love it, though it would be great if the developer would add feature X". There is also a significant difference between a 1 and a 2. For instance, a common meaning of 1 is "it does not work at all for me"; a common meaning of 2 is "it works under some situations, or some beneficial qualities, but overall is very poor".

Your requirement insists that we must throw away this information. That is not a reasonable thing to insist upon.

Added 7/9: Here is another way to look at it.

By your criteria, {1,1,4,4} < {2,2,5,5}, since the median is lower. Now suppose a new user walks by and rates each of these two apps a 3. I would expect this to maintain any existing relationship, so we should have {1,1,3,4,4} < {2,2,3,5,5} -- but your proposal violates this expectation. So the mere act of rating two products, with the same rating, can actually change their relative ranking. That is surprising, to say the least!

To formalize it a bit, I am proposing the following criteria as one that seems reasonable:

• If we have $S < T$, then we also have $S \cup U < T \cup U$ for any set $U$. If we have $S = T$, then we also have $S \cup U = T \cup U$ for any set $U$.

However, this is not consistent with the set of criteria you listed, so we can't have them all: we have to throw something out. I would argue that your third criteria should be thrown out.

-

To answer your question about how to rank the reviews, here is my thought:

I don't share your unhappiness with using the mean. While your arguments may have some merit in principle, in practice I would expect that using the mean (average) rating of all reviews for an app is reasonable and pragmatic.

To address the problem of some apps having very few ratings, you might use Bayesian rating, as explained in the following question: Sorting answers, given overvotes and undervotes. In other words, the corrected rating for an app is a weighted average of (a) the average rating of its reviews, and (b) the average rating taken over all apps. The weights are chosen based upon the number of reviews for this app, so the more reviews it has, the more heavily we weight factor (a).

I would expect that the combination of these two mechanisms would lead to a simple way of ranking apps, that might work well in practice.

-