1
$\begingroup$

Question migrated form https://stackoverflow.com/questions/10019576/confidence-interval-scoring-with-programming-languages#comment12812602_10019576

I used a CI scoring algorithm from http://evanmiller.org/how-not-to-sort-by-average-rating.html, The data is most liked and disliked programming languages of each person. According to the scores it should seem that Python should have a relative higher score, but according to the score data it is in the middle. Am I doing something wrong? Joel Cornett mentioned You need to use Student's T curve, not the normal distribution for smaller sample sizes. Is there a way to fix this?

Here is my gist https://gist.github.com/2305834, which contains the data, scores and the script that generates the scores.

$\endgroup$
  • $\begingroup$ It is correct that scoring by average has problems (due to different variances of the averages), but confidence interval scoring is not the solution. From a statistical standpoint (which seems eminently appropriate for evaluating a purely statistical construct), it is an ad hoc abuse based on a misunderstanding of what CIs do. The financial people have a better, more flexible approach: they "score" based on a bidimensional "frontier" that captures both expected value ("alpha") in one dimension and a measure of uncertainty ("beta") in another. So maybe you're doing nothing wrong at all... $\endgroup$ – whuber Apr 4 '12 at 22:06
1
$\begingroup$

I suggest a much simpler score: (#positives/#negatives) * weight for #respondents (say ln[3+#respondents]).

this gives a more satisfying ranking IMHO:

for  ag   (for/ag).ln[3+for+ag]  language 
3044 0127 193.25 Python
0458 0024 118.01 Clojure
0965 0058 115.36 C
0518 0032 102.23 Haskell
0145 0011 066.82 Lua
1717 0234 055.60 Ruby
0321 0034 055.52 Lisp
0828 0116 048.92 C#
0162 0021 040.31 Erlang
0190 0027 037.96 Scheme
0233 0042 031.22 Scala
0086 0013 030.60 OCaml
0069 0011 027.72 Smalltalk
1411 0433 024.51 JavaScript
0188 0044 023.33 Other
0361 0114 019.54 CoffeeScript
0056 0015 016.07 D
0040 0014 011.55 Forth
0098 0046 010.63 Assembly
0326 0218 009.43 Objective_C
0310 0223 008.74 Perl
0101 0077 006.82 SQL
0529 0564 006.57 C++
0038 0031 005.24 Delphi
0662 1064 004.64 PHP
0024 0020 004.62 Ada
0011 0008 004.25 Rexx
0551 1067 003.82 Java
0032 0037 003.70 Tcl
0023 0025 003.62 Fortan
0066 0097 003.48 Shell
0024 0028 003.43 Pascal
0093 0185 002.83 Actionscript
0036 0091 001.93 ColdFusion
0010 0071 000.62 Cobol
0045 0633 000.46 Visual_Basic
$\endgroup$
  • 2
    $\begingroup$ That weighting looks arbitrary. What is the basis for using it? $\endgroup$ – whuber Apr 5 '12 at 1:09
  • $\begingroup$ The problem statement from the evanmiller article is: "[The average rating algorithm is inadequate because it] puts item two (tons of positive ratings) below item one (very few positive ratings)." ... $\endgroup$ – Riaz Rizvi Apr 5 '12 at 17:24
  • $\begingroup$ ...From this I read we are actually scoring rating and popularity. A high-rated-popular language is far better than a low-rated-unpopular language. With low-rated-popular and high-rated-unpopular languages being conflicted signals.... $\endgroup$ – Riaz Rizvi Apr 5 '12 at 17:24
  • $\begingroup$ ...The ratio is the measure of rating, the weight is my measure of popularity. A log feels like a good reflection of our concept of popularity - incremental increases in popularity are more important in the beginning. ... $\endgroup$ – Riaz Rizvi Apr 5 '12 at 17:25
  • $\begingroup$ ...I am trying to solve for problem statement & Lex Parsimoniae, where a more intricate model has to be able justify its heavy cost of specificity. A simple formula encourages people to tweak it until it reflects people's true perceptions. The middle ground (low-rated-popular vs high-rated-unpopular) is probably subjective and weighting would depend on the target audience. If you were an advocate for new languages you would prefer high-rated-unpopular languages to low-rated-popular ones, vs say a standards maker in a corporation that wants to decide what languages to merge existing systems to. $\endgroup$ – Riaz Rizvi Apr 5 '12 at 17:25
0
$\begingroup$

The article you read isn't a solution as it does not solve the Voter's Paradox. When "voters" are allowed to rank preferences, any mechanism you design will fail to support at least one of the following three properties:

  1. non-dictatorship
  2. absence of irrelevant alternatives
  3. unanimity

The election of both Bill Clinton and Donald Trump are examples of this. In the case of Bill Clinton, most voters preferred George HW Bush to Bill Clinton, so by unanimity, Bush should have won. Additionally, most voters preferred Clinton to Perot and Bush to Perot, so Perot should have no impact. Still Perot drew away enough voters from Bush that Clinton won, even though the majority did not prefer him. Likewise, there is a math theorem that states that if there is a polarizing candidate then they can only come in first or last. Trump meets the mathematical definition of a polarizing candidate, as does Hillary Clinton. They could only come in first or last in their party's nominating systems. Both took first place. Still, both a majority of Republicans and a majority of the population did not want Trump, but there were so many alternative candidates in the Republican party that he won using "first past the post" methods of voting.

You are managing a voting system. You cannot build a good system, but you can build a system that is likely to accomplish your intended goals. What properties do you want to see always present in your ranking system? You choose your solution around the properties that you cannot give up. Cardinal voting may be a future path for you as it is argued that cardinal voting may get around some of these problems. Some Olympic sports use cardinal voting, but there is a body of literature that argues that it isn't the cure people would want.

This is a giant field of study, but the basics are taught in Principles of Microeconomics. I found a PBS lecture on the topic to get you started. You can comment to me for additional help if you need it.

Watch this. It provides five set of "winners" or rankings based on one set of ballots. All that changes is the criterion for what winning means. You are observing the Condorcet paradox. https://www.youtube.com/watch?v=HoAnYQZrNrQ

You need to determine which mathematical properties you want to have present in your system and which you can live without.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.