Method For Calculating Performance from Average Scores

Question

In a simplification of my problem, let's assume we have players who can get an integer value score $s$ considering $\{ s \in \mathbb{N} : 0 \leq s \leq 3 \}$.

Same sample data for scores:

• Player A: average of $2.9$, played $20$ rounds.
• Player B: average of $2.7$, played $60$ rounds.
• Player C: average of $2.2$, played $6$ rounds.
• Player D: average of $2.0$, played $3$ rounds.

I'd like to rank them by performance. Now, even though A has the biggest average, he has much less rounds than B, so the latter still has a better performance. When I thought about how to calculate it, I though about adding a certain small bonus value ($b$) multiplied by the rounds ($r$) to the averages ($a$), and then compare the final result. I think this would work fine. Compare $a + (rb)$.

However, I hit a problem with player D for example. He could have scored $3,3,0$. This gives a low average, but the third result could be just bad luck. Does not imply that he is worse than C who could have a result derived from $2 , 2 , 2, 3 , 2, 2$ for example (consistent lower scores).

How can I precisely calculate that performance score? It should:

• give priority to people with more data
• give less weight to "deviations" than what a simple average does

Answer 1

A couple of key terms that you may want to research that relate are "Standard Error" and "Regression Toward the Mean".

The standard error is a measure of uncertainty that takes sample size into account as well as variability within a group and reflects that fact that we are less uncertain about means based on larger samples.

Regression toward the mean is a somewhat related concept where we tend to see extreme values when looking at individual measurements or means of small groups, then when we observe additional data the new values or means tend to be closer to the overall mean, such as in your example of D seeing a 0 by bad luck and further observations bring D's mean closer to the others.

One approach for analyzing the data would be a Bayesian Hierarchical model (though you need to learn a fair amount to do these properly). This is a method that assumes a certain similarity among your groups and will therefore pull all the means towards a common center, but where you have large samples the means will be pulled a lot less than where you have fewer data points.