How do I sort an ordinal list of user-generated ratings data? The Ubuntu Software Center uses a 1 to 5 star rating system for its App Reviews.  However, it's current ratings sorting algorithm looks very fishy.  I believe this is a different question from this one, which seems to assume the mean of the ratings is a meaningful number.  
If I assume that user supplied ratings are ordinal data, then doing things like adding the ratings together or taking their mean are not correct.  The primary method of sorting must be the sample median.
Unfortunately, this leaves me with a lot of duplicates since 30 applications are being pigeon-holed into 5 stars of ratings, so it must be possible to further subsort the apps with identical median stars.  I believe I want:


*

*Rating at the median should be better than rating below it.


*

*{2,3,3,3,4} > {2,2,3,3,4}


*Similarly, rating at the median should be worse than rating above it.


*

*{2,2,3,3,4} < {2,2,3,4,4}


*Ratings above and below the median should be equivalent.


*

*{1,1,3,4,4} = {2,2,3,4,4} = {2,2,3,5,5}


*Among two apps with the same median, higher confidence that the median is at least that large should rank higher. 


*

*{2,3,3,3,4} > {2,3,4}



Are these reasonable desires?  What algorithm can get me there?
My intuition tells me I want something like sample median + {lower bound probability estimate someone ranks that app higher than median} - {upper bound probability estimate someone ranks that app lower than median}.  So:


*

*a large data set composed of 20 % 1's, 40% 3's, and 40% 4's would approach (3)+(2/5)-(1/5) = 3.2

*a set composed of equal parts 2,3, and 5 would approach (3)+(1/3)-(1/3) = 3.0

*and a set composed of 40% 1's and 60% 3's would approach (3)+(0)-(2/5) = 2.6.


Is this reasonable?
 A: You asked if your requirements look reasonable.  My opinion: No, your requirements are not reasonable.  In particular, you proposed:

Ratings above and below the median should be equivalent.
  
  
*
  
*{1,1,3,4,4} = {2,2,3,4,4} = {2,2,3,5,5}
  

This is not reasonable.  {1,1,3,4,4} is a significantly worse score than {2,2,3,5,5}.
I have studied app market ratings, and found that there is a significant difference between what a 4 vs a 5 (in terms of what this tends to mean to the reviewer).  For instance, a common meaning of 5 is "I love it"; a common meaning of 4 is "I love it, though it would be great if the developer would add feature X".  There is also a significant difference between a 1 and a 2.  For instance, a common meaning of 1 is "it does not work at all for me"; a common meaning of 2 is "it works under some situations, or some beneficial qualities, but overall is very poor".
Your requirement insists that we must throw away this information.  That is not a reasonable thing to insist upon.

Added 7/9: Here is another way to look at it.
By your criteria, {1,1,4,4} < {2,2,5,5}, since the median is lower.  Now suppose a new user walks by and rates each of these two apps a 3.  I would expect this to maintain any existing relationship, so we should have {1,1,3,4,4} < {2,2,3,5,5} -- but your proposal violates this expectation.  So the mere act of rating two products, with the same rating, can actually change their relative ranking.  That is surprising, to say the least!
To formalize it a bit, I am proposing the following criteria as one that seems reasonable:


*

*If we have $S < T$, then we also have $S \cup U < T  \cup U$ for any set $U$.  If we have $S = T$, then we also have $S \cup U = T  \cup U$ for any set $U$.


However, this is not consistent with the set of criteria you listed, so we can't have them all: we have to throw something out.  I would argue that your third criteria should be thrown out.
A: To answer your question about how to rank the reviews, here is my thought:
I don't share your unhappiness with using the mean.  While your arguments may have some merit in principle, in practice I would expect that using the mean (average) rating of all reviews for an app is reasonable and pragmatic.
To address the problem of some apps having very few ratings, you might use Bayesian rating, as explained in the following question: Sorting answers, given overvotes and undervotes.  In other words, the corrected rating for an app is a weighted average of (a) the average rating of its reviews, and (b) the average rating taken over all apps.  The weights are chosen based upon the number of reviews for this app, so the more reviews it has, the more heavily we weight factor (a).
I would expect that the combination of these two mechanisms would lead to a simple way of ranking apps, that might work well in practice.
