Normalizing Data in Multiple Categories

Question

I have a bunch of categories, labeled, say, A-Z. Each of these categories has a bunch of "tasks" inside it. Tasks are specific to a category - they only go into one. For the purposes of abstracting the situation, we'll say there are infinite tasks.

Each category has a different level of difficulty - a task in category A may take 30-40 seconds, whereas a task in category X might take 10-15 minutes.

I have, for the sake of nice numbers, 100 users working in these categories. They choose a pseudo-random category (so you can't rely on their selection of category to normalize things) and may switch multiple times in a day. Some users will only work categories A-C, some D-F, some might work in all of them - it's completely inconsistent.

What I am trying to do is compare these users to each other, to find who is the best/fastest. However, I can't just look at who had the fastest tasks-per-hour, because they might have been doing those tasks in an easy category.

I need to come up with a coefficient for each category to calculate a sort of "weighted total tasks"-per-hour for each user, in order to fairly rank them.

I was previously doing something like:

Total (all users) Completed Tasks in A	Total (all users) time spent in A	(Total (all users) Time Spent in ALL categories	Total (all users) tasks completed in ALL categories)
...	...	...	...

I hope that makes sense. This gave a coefficient (with a mean of all coefficients at '1') for each category that I could use to normalize things.

However, in examining this more closely, I don't think it's fair. If user 23 is really fast, but they only ever work in Category A, that will make everybody who works in category A look bad except for user 23 – user 23 drives the coefficient down.

I think I need a way to normalize a user first to themselves, then normalize the categories coefficients using that user-normalized data. If that makes any sense at all – I'm hurting my OWN brain, here.

Can anybody help me out, or point me in the right direction?

PS: The categories difficulty is changing pseudo-randomly on a live basis, so benchmarking users in each category is a no-go.

Answer 1

This is a really interesting question. I think the way to start is to assign each user a vector, with each entry corresponding to that users speed in a category, and set to 0 (say) if the user hadn't worked in that category. (In your example with categories $A$-$Z$, it would be a 26-dimensional vector.) Then, if two users have worked in the same category, it's straightforward to decide who is better in that category.

But I think you want to go beyond this, and have some overall ranking. Do you know for sure that the same skill set is involved in solving problems from all categories? In other words, if user $x$ is better than user $y$ at category $A$, does that necessarily imply that $x$ is also better at category $B$? If so, then you might be able to rank them, depending on how much overlap there is in categories worked across users.

But in general, this won't be true -- it will be possible that $x$ is better at problems from $A$, but $y$ is better at problems in $B$. Then you have to think about what you mean by "best". Maybe you'll need multiple different notions of "best" for different purposes.

Answer 2

Since your goal is simply to identify the best/fastest user overall, I would start with a mixed/hierarchical model of the form:

lmer(time_spent_on_task ~ task_category + (1|user_id))

This models variation in the time spent on individual tasks as a function of the task categories and user identity. User identity is modelled with a random intercept, taking advantage of partial pooling. Examining the random intercepts would offer one answer to your question: the user with the most negative random intercept would be the estimated fastest user after accounting for variation in time spent due to differences between categories.

That said, this is not a perfect solution and there isn't likely to be one. This is because it's extremely likely that users are not universally faster or slower but differ in how much faster or slower they are based on the type of task (maybe they choose their tasks based on this as well, but you've stipulated that the task choice is pseudo-random and so I will ignore that). In the model, we could account for this by including a random slope that varies with task category. But this does not provide the simpler answer you are looking for - it would mean that the best/fastest user depends on the task category.