Normalizing rating in a group of people [finding effectiveness] I'm a new guy here. Hopefully, I'm asking this question to right forum.
Problem:
We have data of a group of people (P1, P2, P3). They rank their expertise (1-10, where higher number is better) in a list of components (G1, G2, G3).
    P1  P2  P3
--------------
G1 | 8   4   7
G2 | 7   3   7
G3 | 9   6   5

Also, we have some data regarding work done by each person in each component. Example:
For P1,
     W  WD
----------
G1 | 0   0
G2 | 2   0
G3 | 8   2

where W is total work allotted to user P1, and WD is actual work done. W >= WD >= 0.
We have similar data for P2, and P3 users.
Point to note: The user might have some level of expertise regardless of work done in a component. Example: P1 has ranked himself 8/10 even though he has not been given any task in G1 component (W = 0). P1 also has ranked himself 7/10 for G2 even though he has not finished any task in that component (WD = 0).
Now, we want to calculate effective rating of all users relative to the group of users, not self-ranking, considering their work data and self-ranking.
Can anyone suggest some mechanism to achieve this?
Thanks much in advance!
 A: It sounds like you have multiple sources of information about measurements of interest.
The aim is to combine the multiple sources so as to maximise measurement accuracy.
In general you could adopt a formula that weights the different sources of information based on the confidence you ascribe to the source.
It also seems that the confidence in the source might vary.
E.g., the more work data you have on someone, perhaps the more weight this information would be given for that individual.
You may also have evidence on the accuracy of an individual's self ratings.
Thus, a simple strategy would to develop a weighted composite of available information where the size of the weight is relative to the informativeness of the source.
This question on Bayesian rating systems might also be relevant.
A: I'm going to give a stab at an answer, if I misunderstand your question, please comment here or revise your question and I'll do my best to revise my answer.
If I understand you correctly you are trying to find a users 'true level of expertise'.  An elusive quantity to be sure.  Given what you have written, I think you expect that true expertise has some relationship with actual amount of work done.
There are some problems meeting the assumption of the technique I am going to suggest first, but I hope the violations will be tolerable.  A first step could be setting up a regression equation to predict work completed given work assigned for each component (G1, G2, G3, etc).  Residuals from this regression equation are deviations from the normal amount of productivity in that component.  Deviations from normal productivity (if you assume those who are more expert get a greater proportion of their work done) may be viewed as a measure of 'true expertise'.  It also sets each person's productivity in comparison to others (a stated goal).  Of course, I strongly suggest you view a scatter-plot of these data (and compare model fits) as I suspect that the relationship will not be linear and that you'll have have to fit a quadratic.
I'm not sure what you want to do with the self-ratings.  Of course, you should eye-ball the self ratings, if someone is giving themselves all high scores, that participant may need to be dropped. Another thing you might consider doing prior to analysis is subtracting each subject's mean rating of expertise from each of their ratings of expertise. Though in this approach you'll lose information about people who are overall more skilled than others. To validate that the self-ratings are meaningful you want to correlate them with work completed.  They should be somewhat positive given your assumption that those who complete more work on a given component tend to be (but are not necessarily) more expert.  If the self-ratings are shown to be a somewhat valid measure, then you can move on to correlating the self ratings with the above mentioned residual score.  Again, the correlation ought to be somewhat positive.
Edit 1: You say you want to calculate the "effective rating" of all users relative to the group of users, not self-ranking, considering their work data and self-ranking.  I'm not sure how you can both consider self-ranking and not consider it at the same time.  If you mean that you want to create a measure of skill based on something other than self-rankings, but then combine it with the self-ranking data, you can simply sum the Z-score of some measure of skill (e.g. work done or the method I describe above) and the Z-score of self ratings.
A: I worked on it and thought this Bayesian equation will be useful.
RatingTM = SRTM/(1 + AWD/WDTM) + MSRT/(1 + WDTM/AWD)
The variables are:
SR = Team member's self rating
AWD = Average work done by team
WD = Work done by team member
MSRT = Mean self rating of team
(TM: TeamMember)
Please comment if you think this is not right. Thanks!
