What statistical tool can be used to correct for differences in the amount of data an individual is evaluated on?

Question

Let's say an individual gets a score (between 1 and 6) on different pieces of equipment in their department. For example, if I'm proficient at repairing a particular piece of equipment I will score higher, say a 5 or 6. If you then take the average score I get across all the equipment (upwards of 50-100 pieces) in the department in which I work...you get my overall average score. The more effective I am, the more pay I can take home as a result.

The problem I face is that each department has different equipment, so the average is unfair. It may be simpler to gain proficiency (and thus a promotion) in one department simply because they have less equipment. I don't have to improve proficiency on as much equipment so it is mathematically easier. I'd like to include some 'bias' in order to correct for this but I don't know how to do so.

How can you normalize the data despite the difference in sample size?

Seems like an easy question but I'm having difficulty finding the right statistical tool to correct for it. Also, hopefully this is the right place to answer this question.

Answer 1

It seems more like the problem is with your metric. If the average is wiping out an important difference -- then don't use the average!

If someone's value is both a function of their proficiency with a machine, and the number of machines they work on, then why not give them a score based on something like the sum of their proficiencies, within their employing department? That sounds like it would be a good measure of how valuable they were -- to their department.

However, this raises the issue of how many machines an individual can actually maintain, and how much time they spend on each one, so ideally, you would use a payscale related to each level (1-6) and then weight how much of each level they got based on the amount of time they spent on a machine on which they had a certain proficiency -- in other words, I spent 20 hours on a machine where I'm level 3, and 20 hours on a machine where I'm level 5, then I got 1/2 level 3 pay, and 1/2 level 5 pay.

You certainly can normalize the data, despite the different numbers of machines -- the only difference is that you'll be more, or less certain of how accurate your estimate of the average actually is. But from how you describe the problem, it just seems like the average might not be the best route to go down.