Selecting a model for predicting a multinomial response from multinomial data

Question

I am trying to create a predictive model from a set of categorical observations:

task   user
----   ----
S      Alice
M      Bob
M      Alice
S      Charlie
L      Bob
M      Charlie
S      Alice
...    ...

I'm not interested in predicting what type of task will be next for a given user. Rather, I'm interested in predicting a user's most likely distribution of tasks. For example, if a user has a set of 20 tasks, how many of them will be S, M, and L?

The actual dataset has several thousand distinct users that can be grouped hierarchically. Each user belongs to a workgroup, and each workgroup belongs to a department. It's pretty easy to come up with a contingency table and empirical distribution, and I can see that the distribution varies among departments, and among workgroups within each department. I believe that variations among individuals within the same workgroup are insignificant, but this is a statement from intuition, and I don't know how to justify it statistically.

Currently, my predictive model just calculates the distribution of tasks across the entire dataset and uses the same prior distribution for all users: if S/M/L are 0.2/0.5/0.3, then the prediction for every individual is based on this distribution. But in practice, this model doesn't work as well as I would like for several workgroups (and one entire department) because their task distribution appears to be different from that of the whole company. I'm thinking that in these situations, my model needs to use a workgroup-specific or department-specific distribution for the individuals within it. But I would like to avoid over-fitting the model.

So, a few questions:

1. How can I determine if the distribution in a given department or workgroup differs from the total distribution sufficiently to justify using a different set of parameters for that department or workgroup?

2. Let's say I generate a model in which each department has its own distribution. How do I determine whether this model is "better" than the model that uses the same distribution for all departments, or whether the department-specific model is overfitted?

3. Let's say I build a model in which the predictive distribution for an individual is a weighted mixture of the total, the department and the workgroup distributions. How can I figure out the appropriate weights for this mixture?

My dataset is huge (>1M rows), and I have access to Python+SciPy and R. My programming knowledge is much better than my statistical knowledge, so if you describe the statistical technique I can figure out how to implement it in code.

Thanks.

Answer 1

Let's say you were interested in the probability of the next task for a given employee. You could do that with just multiple logit, using the softmax algorithm, with workgroup and employee as grouping, random effects. What you would get out would be a prediction of the most likely distribution of next tasks for an employee in a given work group and department. That distribution is exactly what you want, because it's also the expected distribution of task for the entire workgroup.

So,

1. You will get coefficients for the workgroup and department, and you can use a t-test to see if they are statistically significantly different than 0 for some confidence level.

2. The problem of over-fitting has one common responser, that is given by the name of this website: Cross-validation

3. This will fall out of the softmax estimation.