I couldn't decide if this question belonged here or on the Math stack exchange, but I'm posting it here since it seems more directly relevant.

Basically, I have a bunch of categories, labeled, say, A-Z. Each of these categories has a bunch of "tasks" inside it. Tasks are specific to a category - they only go into one. For the purposes of abstracting the situation, we'll say there are infinite tasks.

Each category has a different level of difficulty - a task in category A may take 30-40 seconds, whereas a task in category X might take 10-15 minutes.

I have, for the sake of nice numbers, 100 users working in these categories. They choose a pseudo-random category (so you can't rely on their selection of category to normalize things) and may switch multiple times in a day. Some users will only work categories A-C, some D-F, some might work in all of them - it's completely inconsistent.

What I am trying to do is compare these users to eachother, to find who is the best/fastest. However, I can't just look at who had the fastest tasks-per-hour, because they might have been doing those tasks in an easy category.

I need to come up with a coefficient for each category to calculate a sort of "weighted total tasks"-per-hour for each user, in order to fairly rank them.

I was previously doing something like:

(Total (all users) Completed Tasks in A/Total (all users) time spent in A)/(Total (all users) Time Spent in ALL categories/Total (all users) tasks completed in ALL categories)

I hope that makes sense. This gave a coefficient (with a mean of all coefficients at '1') for each category that I could use to normalize things.

However, in examining this more closely, I don't think it's fair. If user 23 is really fast, but they only ever work in Category A, that will make everybody who works in category A look bad except for user 23 - user 23 drives the coefficient down.

I think I need a way to normalize a user first to themselves, then normalize the categories coefficients using that user-normalized data. If that makes any sense at all - I'm hurting my OWN brain, here.

Can anybody help me out, or point me in the right direction?

PS: The categories difficulty is changing pseudo-randomly on a live basis, so benchmarking users in each category is a no-go.


This is a really interesting question. I think the way to start is to assign each user a vector, with each entry corresponding to that users speed in a category, and set to 0 (say) if the user hadn't worked in that category. (In your example with categories $A$-$Z$, it would be a 26-dimensional vector.) Then, if two users have worked in the same category, it's straightforward to decide who is better in that category.

But I think you want to go beyond this, and have some overall ranking. Do you know for sure that the same skill set is involved in solving problems from all categories? In other words, if user $x$ is better than user $y$ at category $A$, does that necessarily imply that $x$ is also better at category $B$? If so, then you might be able to rank them, depending on how much overlap there is in categories worked across users.

But in general, this won't be true -- it will be possible that $x$ is better at problems from $A$, but $y$ is better at problems in $B$. Then you have to think about what you mean by "best". Maybe you'll need multiple different notions of "best" for different purposes.

  • $\begingroup$ Thanks for the detail (I'm the same as the OP, my account's giving me issues)! I know the same skillset is required for each category, so that's not a problem. And while your suggestion does help me rank the users, do you know of a way to effectively generate a 'weight' - an actual, numerical coefficient - that I can calculate for each category? Effectively to say "3 tasks in category A are worth 5 in category b" because I'll have a conversion rate. That's what I'm aiming for. $\endgroup$
    – Helpful
    Apr 22 '14 at 10:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.