How do I estimate the probability P(skill(player1)>skill(player2)) when all of the following apply:
- I know the number of wins and losses of player1 and player2.
- I can assume that the wins and losses are all against randomly drawn opponents from the same population for each of player1 and player2.
- Equality is out of the question.
- The skill measure is a relative position in the list of all players. The skill measure is now to be treated as a random variable with some distribution that depends on the wins and losses of the player measured.
I need this for improving a compare-sort algorithm where I am constantly trying to choose the comparison that, through transitive closure, gives the outcome where fewest further comparison actions are needed. Since a comparison can lead to two results, I will weight the possible outcomes by their estimated probability, in order to make better choices of comparisons. I do not care too much about computational efficiency, as there will be relatively small datasets involved.