I'm trying to solve the following problem. Multiple people score multiple vendors. The problem is that some people are less critical than others which results in them giving higher scores in general. Both #people and #vendors is low. Thus if you just sum all the scores for one vendor, the person being more generous has a bigger impact on the outcome.

The strategy I applied first was:

  1. calculate z score within one person's scores
  2. Map to 0 - 1 range within person's scores
  3. Divide each score of person by sum of his scores.

This way I keep the proportionality of their scores and the total "points" they gave is 1, making it equal for everyone.

Is this the correct way to tackle this problem or not?


If I understand correctly, you are trying to get a mean score for each vendor and you have multiple raters (people) who rate the vendors, but their scores are unreliable (because they consistently underrate or overrate a vendor).

In that case one option would be to z-score (standardization) the raters' scores (not the vendors'), such that all scores for each rater are on the same scale. So if a rater consistently assigned higher scores than usual, this higher score will become his "normal" score - his average.

After that you can estimate a mean z-score for each vendor. This will change the interpretation of the score. What you will get is a measure of which vendors are above average (>0) and which are below average (<0), and the magnitude will tell you by how much they are far from average (based on the raters' opinion).

| cite | improve this answer | |
  • $\begingroup$ Yes that is exactly the problem. I think that's what I did? Except I put that on a different scale? After step 1 I just translated the z score and scaled it to the range of [0..1] and then normalized divided it for one person by the total amount of scoring they did so they in total only gave one point. This would give the same results, no? $\endgroup$ – Spyral Feb 6 at 14:15
  • $\begingroup$ @Spyral It's not exactly the same, I would say your additional steps are unnecessary. $\endgroup$ – user2974951 Feb 7 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.