I have looked everywhere and I am struggling to find a solution. Granted, I am no statistician but I do have some knowledge of basic statistics. Any help is truly appreciated!
There are a number of employees being judged by a committee on certain competencies. The judges rate each employee from 1 to 5 for each competency. Then all of the competency scores for the employee are added up to get their total score from that particular judge. After I receive all of the total scores from each judge, I calculate the mean to get their total score. This calculated total score is used to compare them/rank them against other employees.
Here is my problem though: some of the judges may not know the employee. When they don't know them, they are not to score the employee at all, leaving room for inconsistency.
For example, Bob might be scored by all 10 judges with a mean of 17.5, but then Jill only has 1 judge who knows her and they score her at 25, which would be her mean score. This does not seem fair to Bob as Jill just had 1 good score. Is there a mathematical way to even the playing field so that I can rank employees in the most consistent manner?