I couldn't decide if this question belonged here or on the Math stack exchange, but I'm posting it here since it seems more directly relevant.
Basically, I have a bunch of categories, labeled, say, A-Z. Each of these categories has a bunch of "tasks" inside it. Tasks are specific to a category - they only go into one. For the purposes of abstracting the situation, we'll say there are infinite tasks.
Each category has a different level of difficulty - a task in category A may take 30-40 seconds, whereas a task in category X might take 10-15 minutes.
I have, for the sake of nice numbers, 100 users working in these categories. They choose a pseudo-random category (so you can't rely on their selection of category to normalize things) and may switch multiple times in a day. Some users will only work categories A-C, some D-F, some might work in all of them - it's completely inconsistent.
What I am trying to do is compare these users to eachother, to find who is the best/fastest. However, I can't just look at who had the fastest tasks-per-hour, because they might have been doing those tasks in an easy category.
I need to come up with a coefficient for each category to calculate a sort of "weighted total tasks"-per-hour for each user, in order to fairly rank them.
I was previously doing something like:
(Total (all users) Completed Tasks in A/Total (all users) time spent in A)/(Total (all users) Time Spent in ALL categories/Total (all users) tasks completed in ALL categories)
I hope that makes sense. This gave a coefficient (with a mean of all coefficients at '1') for each category that I could use to normalize things.
However, in examining this more closely, I don't think it's fair. If user 23 is really fast, but they only ever work in Category A, that will make everybody who works in category A look bad except for user 23 - user 23 drives the coefficient down.
I think I need a way to normalize a user first to themselves, then normalize the categories coefficients using that user-normalized data. If that makes any sense at all - I'm hurting my OWN brain, here.
Can anybody help me out, or point me in the right direction?
PS: The categories difficulty is changing pseudo-randomly on a live basis, so benchmarking users in each category is a no-go.