# Methodology to rank contributors

I have contributors who complete certain no of tasks in a day. I want to rank them. Below is the data.

User 1 has been given 1 task and he completed it = 100% completion

User 2 has been given 20 tasks and he completed 17 = 85% completion

I want to rank user 2 higher than user 1 as it is considerably difficult to maintain 100% completion as the no of tasks given increase.

Sample data:

+------+-------+-----------+------+-----+
|      |   1   |     2     |  3   |  4  |
+------+-------+-----------+------+-----+
| User | Given | Completed | %    | 2*3 |
| 1    | 1     | 1         | 100% | 1   |
| 2    | 5     | 1         | 20%  | 0   |
| 3    | 19    | 6         | 32%  | 2   |
| 4    | 19    | 0         | 0%   | 0   |
| 5    | 16    | 2         | 13%  | 0   |
| 6    | 13    | 1         | 8%   | 0   |
| 7    | 17    | 2         | 12%  | 0   |
| 8    | 12    | 4         | 33%  | 1   |
| 9    | 20    | 17        | 85%  | 14  |
| 10   | 6     | 2         | 33%  | 1   |
| 11   | 15    | 10        | 67%  | 7   |
| 12   | 14    | 0         | 0%   | 0   |
| 13   | 5     | 4         | 80%  | 3   |
| 14   | 7     | 4         | 57%  | 2   |
| 15   | 12    | 5         | 42%  | 2   |
| 16   | 17    | 10        | 59%  | 6   |
| 17   | 17    | 1         | 6%   | 0   |
| 18   | 18    | 11        | 61%  | 7   |
| 19   | 9     | 3         | 33%  | 1   |
| 20   | 20    | 20        | 100% | 20  |
+------+-------+-----------+------+-----+


Below is the graphical representation of the sample data.

Graph:

Q1 - who fall in Q1 are completed more than 10 tasks and % completion is > 50%.

Q2 - who fall in Q2 are completed more than 10 tasks but % completion is < 50%

Q3 - who fall in Q3 are completed less than 10 tasks but % completion is > 50%

Q4 - who fall in Q4 are completed less than 10 tasks but % completion is > 50%

Roughly the ideal ranking order should be Q1 > Q2 ~ Q3 > Q4. I am still undecided on if Q2 > Q3 or Q3 > Q4.

My Approach so far I have multiplied No of tasks completed * % completion to get the score of the user. This is to given user who complete more tasks higher rank.

But there are some obervation where this logic doesn't look good. User 3 has falls in Q4 and has better score then user 1 who fall in Q1.

2 questions:

1. which should I rate better ? Q3 or Q2 ( i know it's subjective but suggestion welcome)
2. What is a statistically robust method to implement my logic ?
• You can estimate the overall probability of completion. Then estimate the probability of each person getting their results or better just by chance. Ranking is then on the inverse of that P-value. – Nick Cox Dec 16 '15 at 13:24
• I don't understand why you want me to estimate probability. Is the question unclear ? or am i missing something ? – user98374 Dec 16 '15 at 17:46
• The central question is who did well and the argument is, in your example, that 1/1 deserves less credit than 17/20, i.e. that 1/1 is more consistent with chance (or luck or good fortune) than 17/20. Therefore you can, given some overall probability of success (which can only be estimated from the data, so far as I can see on your information), estimate tail probabilities for each person on a binomial model. Then rank accordingly. – Nick Cox Dec 16 '15 at 18:02
• It's a little like evaluating magicians who claim to be able to produce heads every time on tossing a coin. 1 head out of 1 is less convincing than 17 out of 20. That's your stance, and I agree, and I am underlining that it's a standard problem. – Nick Cox Dec 16 '15 at 18:05

The central question is ranking how well people did. The argument is, in your example of users 1 and 2, that 1/1 deserves less credit than 17/20, i.e. that 1/1 is more consistent with chance (or luck or good fortune) than 17/20. Therefore you can, given some overall probability of success (which can only be estimated from the data, so far as I can see on your information), estimate tail probabilities for each person on a binomial model. Then rank accordingly.

It's a little like evaluating magicians judged on their ability to produce heads every time on tossing a coin. 1 head out of 1 is less convincing than 17 out of 20. That's your stance, and I agree. It can be interpreted as a standard problem.

Substantively, it remains possible that the number of completions provides a better measure.

Here is a worked example with Stata code. Code in any other software should be simple. Note that binomialtail($n,k,p$ ) is here the probability of observing $k$ or more successes in $n$ trials when the probability of a success on one trial is $p$. I am estimating $p$ as 104/262. Stata by default ranks lowest value as 1 and given ties splits ranks to preserve the sum of the ranks.

clear
input User  Given  Completed
1     1      1
2     5      1
3     19     6
4     19     0
5     16     2
6     13     1
7     17     2
8     12     4
9     20     17
10    6      2
11    15     10
12    14     0
13    5      4
14    7      4
15    12     5
16    17     10
17    17     1
18    18     11
19    9      3
20    20     20
end
su Given , meanonly
di r(sum)
su Completed, meanonly
di r(sum)
gen P = binomialtail(Given, Completed, =104/262')
egen Rank = rank(P)
sort Rank
l User Given Completed P Rank

+-------------------------------------------+
| User   Given   Comple~d          P   Rank |
|-------------------------------------------|
1. |   20      20         20   9.43e-09      1 |
2. |    9      20         17   .0000421      2 |
3. |   11      15         10   .0320042      3 |
4. |   18      18         11   .0545732      4 |
5. |   13       5          4   .0847158      5 |
|-------------------------------------------|
6. |   16      17         10   .0876133      6 |
7. |   14       7          4   .2839052      7 |
8. |    1       1          1   .3969466      8 |
9. |   15      12          5    .553135      9 |
10. |   10       6          2   .7619411     10 |
|-------------------------------------------|
11. |   19       9          3   .7624251     11 |
12. |    8      12          4   .7681059     12 |
13. |    3      19          6   .8303364     13 |
14. |    2       5          1   .9202411     14 |
15. |    5      16          2   .9964715     15 |
|-------------------------------------------|
16. |    7      17          2   .9977507     16 |
17. |    6      13          1   .9986048     17 |
18. |   17      17          1   .9998155     18 |
19. |    4      19          0          1   19.5 |
20. |   12      14          0          1   19.5 |
+-------------------------------------------+
`
• Perfect. This is really helpful. – user98374 Dec 17 '15 at 5:07
• This concept worked great and I want to extend this a little further, should I ask here or create a new question ? – user98374 Jan 8 '16 at 6:08
• Ask a new question. Give a cross-reference to this thread – Nick Cox Jan 8 '16 at 10:15