# Sorting a list of averages

This might sound like a tech question at first, but there's a statistical logic that I'm trying to come up with. Help would be really appreciated.

Context: At my organization, we have a custom-built webapp where people add their skills (Project Management, Java, Cooking, etc) and other people in the organization rate those skills on a scale of 1 to 5 stars (no decimal values). Anyone in the organization can rate anyone's skills. So my average rating (which could be a decimal value) for a skill is: (the ratings I've received)/(total number of ratings). And each person can rate my skills only once, so if they want, they can change the original rating that they gave me on a certain skill. There's an administrator view, where the HR or the CEO can view a list of all the skills that the organization has, the number of people who possess a certain skill, and the ranking of people against that given skill.

Scenario: Up until now, we have sorted (in descending order) this list of people by the average rating for each person. But there's a problem with that. Someone who has received only one 5-star rating is ranked higher than someone who has received 20 4-star ratings (and the latter is a more skilled person). Switching the ranking criteria to number of ratings won't help either, because this way a person with 50 1-star ratings (hence a lesser skilled person) would be ranked higher than someone with 4 5-star ratings.

Question: We need one list, sorted in a way that we're able to find the most skilled person. Someone who's not only rated higher, but also the one with more ratings. What would be the logic that we could use to have a sorted list which puts the more skilled person on top of one who's lesser skilled?

• On the top of my head I'd probably use score(avg, n) = avg * (1 - 1/(n+1)) where avg is the average score and n is the amount of points for that person. Commented Jan 25, 2018 at 12:49
• (by which I meant data points/ratings) Commented Jan 25, 2018 at 12:55
• Do the skills job related/ specific? Or for KPI? I mean, cook is good at cooking: 5-star = very yummy Manager is good at management/ team management/ etc: 5-star = very good management skill IT person is good at programming/ project management/ etc: 5-star = very good programming 5-star = very good project management Can a person having >= 1 skills? Commented Jan 25, 2018 at 13:18
– Tim
Commented Jan 25, 2018 at 13:23

2. Shrink the ratings towards average (or other value) by weighting them by their counts. If the average rating that $i$-th person got was $m_i$ and she was rated by $n_i$ people, and $\mu$ is the global average of the ratings, then create a shrinked rating by
$$\tilde m_i = \frac{m_i n_i + \mu \alpha}{n_i + \alpha}$$
where $\alpha$ is the "pseudocount" of the $\mu$ rating. So if $n_i < \alpha$, then the rating is closer to $\mu$ then $m_i$, if $n_i = \alpha$, you take their average, and if $n_i > \alpha$ then the ratings gets closer to $m_i$. People sometimes call it the "Bayesian average" but the name is quite misleading.