Creating a rank on performance and size

Question

Ranking is not something that I need to do particularly frequently in my job and its probably the weakest part of my statistics background. I have a dataset containing a list of items and a performance metric associated with each (a percentage success). Items in this list have always been ranked based upon that percentage, but I have noticed that items with small size bounce around the ranking on a weekly basis (the cost of 1 failure for a small sized item shifts the percentage hugely and results in an item jumping from 10th to 500th). Similarly there are a number of small items that often top the ranking (yay great, you got 4 of 4 right! while the item in second got 498 of 500 right...). I want to create a new ranking function that combines the quality metric with the size of the item over which the quality metric was calculated. Any guidance on a standard approach to this would be very helpful.

Answer 1

Rankings are the stuff statistical nightmares are made of. The reason for saying this is that once one gets beyond simple univariate data there is rarely, if ever, a "ground truth" on which to build an infallibly accurate, incontestable solution. Read Malcolm Gladwell's excellent New Yorker article on the kluge-y nature of college rankings, The Order of Things, to get a sense of just how fraught rankings really are.

http://www.newyorker.com/magazine/2011/02/14/the-order-of-things

Caveats such as are expressed in Gladwell's article never stop people from developing rankings since few ever penetrate the foggy mists of method and ask the tough questions about what it means and is really doing.

So, what can one do in your case? The important thing to bear in mind is that exclusive use of and reliance on relative metrics -- such as percentages -- biases the ranking in one direction. A simple corrective is, as noted by Evad in his answer, to add a second metric based on the absolute values of success. By combining relative and absolute metrics, one can hope to arrive at something like a normalized ranking.

However, there are lots of ways of combining things and it's here that the rubber really meets the road, so to speak. The key questions to ask at this point are Who is the audience for this metric? and If I do something sophisticated, will they understand it? Here are some possible approaches to combining these metrics:

• Create deciles for each, add the values of the resulting decile assignments and rank that

• Since they are in differing units, standardize each to a mean of zero and std dev of one, add them together and rank that

• Use principal components to combine them into a single latent vector and rank that

The things to bear in mind are, first, that the possibilities are almost endless in terms of the heuristics that can be created from such a seemingly innocent and simple stream of information and, second, that there is no one "right" or true way to bring them together to create a rank ordering.

Answer 2

A simple method would be to take the average success rate weighted with actual success rate of an item [where the total trials for an item is low eg below 10, or below 15. In a nice scenario the average success rate for every item would be in theory be 50% if left to chance and normally distributed.

Any threshold can be used to start with but it will be better to find the statistically significant point where 4 out of 4, or 7 out of 7 is very unlikely to have happened by chance [if there are 500+ items, 9 or 10 probably will be the minimum starting point for average 50% and may be higher / lower dependent on the average probability of success]. The threshold must be kept low though, or this method could mute statistically significant items [20 out of 20, with 50% expected success].

Based on a threshold of 10:

• If trials < 10: (Trials/Threshold*SuccessRate) + (Threshold-Trials)*Average success rate
• If trials >=10: Use actual success rate.

Example:

• If an item has a success rate of 8 out of 8 [100%]. Then due to low confidence as it possibly occurred entirely by chance, it is averaged with weighted average success rate.
• Now if the success for the item continues, line 2 shows the new confidence based expected success rate.

• Finally if the success continues and reaches 10/10 [line 3], the average success of item is nolonger weighted with actual success rate. This requires programming / formulas to avoid using the formula once 10+ trials are reached and you are confident in success rate.

  8 out of 8: 8/10 * 8/8 + (10-8) * 50% = 90%

  9 out of 9: 9/10 * 9/9 + (10-9) * 50% = 95%

  10 out of 10: 10/10 * 10/10 + (10-10) * 50% = 100%

  11 out of 11: 11/11 = 100% *Use actual success rate

This method can be used in an exponential manner as well [which will add considerable improvement]. An even simpler method is to simply exclude items below the threshold from being ranked.

Important: Items likely will not naturally all have same expected success rates which needs to be accounted for in any method unless the list is for the most part arbitrary. Example: comparing sales success of an inbound call centre verse a telemarketer salesperson.

The average salesperson might have an expected success rate of 50% [average of all sale staff] for a customer calling who is already interested, so a success rate of 16/20 [80%] from is great. BUT a telemarketer might have an expected success rate of 5% who achieves a success rate of 5/20 [25%] is actually a slightly better salesperson than the call centre worker and if a list were to be made for all phone based salespersons between the two companies, that telemarketer would be unjustly ranked very poorly in such a list.