0
$\begingroup$

This might sound like a tech question at first, but there's a statistical logic that I'm trying to come up with. Help would be really appreciated.

Context: At my organization, we have a custom-built webapp where people add their skills (Project Management, Java, Cooking, etc) and other people in the organization rate those skills on a scale of 1 to 5 stars (no decimal values). Anyone in the organization can rate anyone's skills. So my average rating (which could be a decimal value) for a skill is: (the ratings I've received)/(total number of ratings). And each person can rate my skills only once, so if they want, they can change the original rating that they gave me on a certain skill. There's an administrator view, where the HR or the CEO can view a list of all the skills that the organization has, the number of people who possess a certain skill, and the ranking of people against that given skill.

Scenario: Up until now, we have sorted (in descending order) this list of people by the average rating for each person. But there's a problem with that. Someone who has received only one 5-star rating is ranked higher than someone who has received 20 4-star ratings (and the latter is a more skilled person). Switching the ranking criteria to number of ratings won't help either, because this way a person with 50 1-star ratings (hence a lesser skilled person) would be ranked higher than someone with 4 5-star ratings.

Question: We need one list, sorted in a way that we're able to find the most skilled person. Someone who's not only rated higher, but also the one with more ratings. What would be the logic that we could use to have a sorted list which puts the more skilled person on top of one who's lesser skilled?

$\endgroup$
  • $\begingroup$ On the top of my head I'd probably use score(avg, n) = avg * (1 - 1/(n+1)) where avg is the average score and n is the amount of points for that person. $\endgroup$ – mroman Jan 25 '18 at 12:49
  • $\begingroup$ (by which I meant data points/ratings) $\endgroup$ – mroman Jan 25 '18 at 12:55
  • $\begingroup$ Do the skills job related/ specific? Or for KPI? I mean, cook is good at cooking: 5-star = very yummy Manager is good at management/ team management/ etc: 5-star = very good management skill IT person is good at programming/ project management/ etc: 5-star = very good programming 5-star = very good project management Can a person having >= 1 skills? $\endgroup$ – Snowman Jan 25 '18 at 13:18
  • $\begingroup$ @Snowman please do not use answers for commenting the questions. $\endgroup$ – Tim Jan 25 '18 at 13:23
0
$\begingroup$

The two simple solutions that come to my mind are:

  1. Choose a threshold for minimal number of votes that you consider as valid and do not consider the people who didn't reach the threshold in your competition.

  2. Shrink the ratings towards average (or other value) by weighting them by their counts. If the average rating that $i$-th person got was $m_i$ and she was rated by $n_i$ people, and $\mu$ is the global average of the ratings, then create a shrinked rating by

    $$ \tilde m_i = \frac{m_i n_i + \mu \alpha}{n_i + \alpha} $$

    where $\alpha$ is the "pseudocount" of the $\mu$ rating. So if $n_i < \alpha$, then the rating is closer to $\mu$ then $m_i$, if $n_i = \alpha$, you take their average, and if $n_i > \alpha$ then the ratings gets closer to $m_i$. People sometimes call it the "Bayesian average" but the name is quite misleading.

    So you basically say that if you don't have enough votes, then you consider those employees as "average".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.