Confidence interval scoring Question migrated form https://stackoverflow.com/questions/10019576/confidence-interval-scoring-with-programming-languages#comment12812602_10019576
I used a CI scoring algorithm from http://evanmiller.org/how-not-to-sort-by-average-rating.html, The data is most liked and disliked programming languages of each person. According to the scores it should seem that Python should have a relative higher score, but according to the score data it is in the middle. Am I doing something wrong? Joel Cornett mentioned You need to use Student's T curve, not the normal distribution for smaller sample sizes. Is there a way to fix this?
Here is my gist https://gist.github.com/2305834, which contains the data, scores and the script that generates the scores.
 A: The article you read isn't a solution as it does not solve the Voter's Paradox.  When "voters" are allowed to rank preferences, any mechanism you design will fail to support at least one of the following three properties:


*

*non-dictatorship

*absence of irrelevant alternatives

*unanimity


The election of both Bill Clinton and Donald Trump are examples of this.  In the case of Bill Clinton, most voters preferred George HW Bush to Bill Clinton, so by unanimity, Bush should have won.  Additionally, most voters preferred Clinton to Perot and Bush to Perot, so Perot should have no impact.  Still Perot drew away enough voters from Bush that Clinton won, even though the majority did not prefer him.  Likewise, there is a math theorem that states that if there is a polarizing candidate then they can only come in first or last.  Trump meets the mathematical definition of a polarizing candidate, as does Hillary Clinton.  They could only come in first or last in their party's nominating systems. Both took first place.  Still, both a majority of Republicans and a majority of the population did not want Trump, but there were so many alternative candidates in the Republican party that he won using "first past the post" methods of voting.
You are managing a voting system.  You cannot build a good system, but you can build a system that is likely to accomplish your intended goals.  What properties do you want to see always present in your ranking system?  You choose your solution around the properties that you cannot give up.  Cardinal voting may be a future path for you as it is argued that cardinal voting may get around some of these problems.  Some Olympic sports use cardinal voting, but there is a body of literature that argues that it isn't the cure people would want.
This is a giant field of study, but the basics are taught in Principles of Microeconomics.  I found a PBS lecture on the topic to get you started.  You can comment to me for additional help if you need it.
Watch this. It provides five set of "winners" or rankings based on one set of ballots.  All that changes is the criterion for what winning means.  You are observing the Condorcet paradox.
https://www.youtube.com/watch?v=HoAnYQZrNrQ
You need to determine which mathematical properties you want to have present in your system and which you can live without.
