"normalising" scores for performance First time poster, have had a browse of the open question forum but cant find an answer to my problem... I dont know if "normalising" is the correct term here... so be gentle... :-)
I am required to generate a "score" for a team of managers, based around a list of different criteria.
Each criteria ultimately gives a reasonably convenient score that can be expressed as a %
So far so good. I had thought I could take an average of the % and be done.
Key issue / problem for me here is that, of a total of 8 or so criteria, not all come in to play for each manager. So, for managers with less criteria in play, any low scores have a greater impact than for managers with more criteria in play.
For example:
Manager a
criteria 1: 100%
Criteria 2: 100%
Criteria 3: 75%
Criteria 4: 100%
Average score: 93.75%
Manager b
criteria 1: 100%
Criteria 2: 100%
Criteria 3: 75%
Criteria 4: n/a
Average score: 91.66%
Ultimately meaning, manager b is more negatively impacted for making the same error. This scoring system will be used to generate bonus scores - and its a hugely contentious topic.
What options do I have for correcting this without giving "manager b" a free 100% for criteria 4?
Thanks in advance!
 A: You could populate any NAs with the average score achieved by the other managers, either at a total level or criteria level 
At a total level it would be
Manager b criteria 1: 100% Criteria 2: 100% Criteria 3: 75% Criteria 4: 93.75% Average score: 92.18%.
At a criteria level it would be
Manager b criteria 1: 100% Criteria 2: 100% Criteria 3: 75% Criteria 4: 100% Average score: 93.75%.
A: Since this is a hugely contentious topic you may have to use more simple/less rigorous mathematics to allow people to understand what you're doing.
You're right to be skeptical of just averaging the criteria percentages, in your example it looks like it's easy to score high in criteria 4 so manager B is being assessed in fewer of the easy criteria.
Normalization is done by subtracting the mean and dividing by the standard deviation of the scores for that criteria. If $X$ is the criteria score with mean $\mu$ and standard deviation $\sigma$ then the normalized criteria score $Y$ is given by
$$Y=\frac{X-\mu}{\sigma}$$
Normalizing is useful because it accounts for how high the average score is by subtracting the mean (this fixes the problem highlighted for manager B missing out on criteria 4), it also takes account of how spread out the scores are by dividing by the standard deviation.
Here's an example to show why dividing by the standard deviation is important.
Suppose you have five managers who are assessed on a criteria and get scores of $50\%, 40\%, 60\%, 65\%, 35\%$. The average is 50% so the manager who got 35% was 15 percentage points below the average.
Now suppose they are assessed on another criteria and their scores are $50\%, 30\%, 70\%, 80\%, 20\%$. The average is still 50% but now everybody's deviation from 50% is double what it was in the previous criteria. The manager who got 20% is 30 percentage points below the average but since everybody is twice as far from the average in this criteria a 30 percentage point difference in the second criteria is equivalent to a 15 percentage point difference in the first criteria.
Dividing by the standard deviation fixes this problem.
If you take this approach and normalize all the criteria scores you can average those normalized scores to get a fairer overall assessment of the managers' performance which won't penalize or favour somebody for not being assessed in one criteria.
