Normalization of a 360-degree feedback evaluation

Question

First of, I'm not a data scientist or anything like that so there will be a lot of hand waving (sorry about that), I just hope to get the gist across.

At my company we're doing a 360-degree feedback evaluation process. It goes a little something like this:

• we have categories, C1, C2, ..., Cn
• we have people, P1, P2, ..., Pm
• each person P will create a matrix [n, m] in which they'll grade all other people in a range 1-5

The output would be like this (for P2):

   C1 C2 C3 C4 C5
-----------------
P1  4  2  3  4  4
P2  -  -  -  -  -  # (does not evaluate self)
P3  4  3  1  5  5
P4  2  2  2  2  2

What I'd like to do is calculate a sort of "meritocratic average grades for person P" with the following hypothesis:

• a person P is capable (not capable) in category C
• a grade that person gives in C is more (less) valuable than someone else who is more average in category C (as P knows what he/she is talking about)
• I'd like to figure out the ranking based on normalized grades

So, what I got so far is to average the grades for C and figure out the "category worth" of P, call it CW. Then, all his grades are to be pondered with CW, but this way people with lower CW will give even lower grades.

This seems to go against the idea of having them have less influence to the final grade (and will lower the overall grade for person P).

Am I missing something here?

Answer 1

What about the following iterative procedure to evaluate a person's grade in category C:

• At the initial state $k = 0$ each person $i$ is given the initial grade $P_i^0$ which is the simple average of the grades $G(j, i)$ given by person $j$

$$ P_i^0 = \frac{\sum_{j \ne i}G(j, i)}{\sum_{j \ne i}1}. $$

• At step $k$ we give more weight to $G(j, i)$ by multiplying by the $k-1$ grade of person $j$. That is

$$ P_i^k = \frac{\sum_{j \ne i}P_j^{k-1}G(j, i)}{\sum_{j \ne i}P_j^{k-1}}. $$

• You iterate over $k$ until convergence.