# Confidence interval scoring

I used a CI scoring algorithm from http://evanmiller.org/how-not-to-sort-by-average-rating.html, The data is most liked programmings and most disliked programming languages of each person. According to the scores it should seem that Python should have a relative higher score, but according the the score data it is in the middle. Am I doing something wrong? Joel Cornett mentioned You need to use Student's T curve, not the normal distribution for smaller sample sizes. Is there a way to fix this?

Here is my gist https://gist.github.com/2305834, which contains the data, scores and the script that generates the scores.

-
It is correct that scoring by average has problems (due to different variances of the averages), but confidence interval scoring is not the solution. From a statistical standpoint (which seems eminently appropriate for evaluating a purely statistical construct), it is an ad hoc abuse based on a misunderstanding of what CIs do. The financial people have a better, more flexible approach: they "score" based on a bidimensional "frontier" that captures both expected value ("alpha") in one dimension and a measure of uncertainty ("beta") in another. So maybe you're doing nothing wrong at all... –  whuber Apr 4 '12 at 22:06

I suggest a much simpler score: (#positives/#negatives) * weight for #respondents (say ln[3+#respondents]).

this gives a more satisfying ranking IMHO:

for  ag   (for/ag).ln[3+for+ag]  language
3044 0127 193.25 Python
0458 0024 118.01 Clojure
0965 0058 115.36 C
0145 0011 066.82 Lua
1717 0234 055.60 Ruby
0321 0034 055.52 Lisp
0828 0116 048.92 C#
0162 0021 040.31 Erlang
0190 0027 037.96 Scheme
0233 0042 031.22 Scala
0086 0013 030.60 OCaml
0069 0011 027.72 Smalltalk
1411 0433 024.51 JavaScript
0188 0044 023.33 Other
0361 0114 019.54 CoffeeScript
0056 0015 016.07 D
0040 0014 011.55 Forth
0098 0046 010.63 Assembly
0326 0218 009.43 Objective_C
0310 0223 008.74 Perl
0101 0077 006.82 SQL
0529 0564 006.57 C++
0038 0031 005.24 Delphi
0662 1064 004.64 PHP
0011 0008 004.25 Rexx
0551 1067 003.82 Java
0032 0037 003.70 Tcl
0023 0025 003.62 Fortan
0066 0097 003.48 Shell
0024 0028 003.43 Pascal
0093 0185 002.83 Actionscript
0036 0091 001.93 ColdFusion
0010 0071 000.62 Cobol
0045 0633 000.46 Visual_Basic

-
That weighting looks arbitrary. What is the basis for using it? –  whuber Apr 5 '12 at 1:09
The problem statement from the evanmiller article is: "[The average rating algorithm is inadequate because it] puts item two (tons of positive ratings) below item one (very few positive ratings)." ... –  Riaz Rizvi Apr 5 '12 at 17:24
...From this I read we are actually scoring rating and popularity. A high-rated-popular language is far better than a low-rated-unpopular language. With low-rated-popular and high-rated-unpopular languages being conflicted signals.... –  Riaz Rizvi Apr 5 '12 at 17:24
...The ratio is the measure of rating, the weight is my measure of popularity. A log feels like a good reflection of our concept of popularity - incremental increases in popularity are more important in the beginning. ... –  Riaz Rizvi Apr 5 '12 at 17:25
...I am trying to solve for problem statement & Lex Parsimoniae, where a more intricate model has to be able justify its heavy cost of specificity. A simple formula encourages people to tweak it until it reflects people's true perceptions. The middle ground (low-rated-popular vs high-rated-unpopular) is probably subjective and weighting would depend on the target audience. If you were an advocate for new languages you would prefer high-rated-unpopular languages to low-rated-popular ones, vs say a standards maker in a corporation that wants to decide what languages to merge existing systems to. –  Riaz Rizvi Apr 5 '12 at 17:25