I am designing an experiment to measure the effects of two different workflow modifications, individually or combined, on overall group productivity. The null hypothesis is that the modifications make no difference. But, the modifications could also help or hurt productivity. The workflow modifications pertain to altering the order in which tasks are handled and to grouping tasks with similar characteristics together.

Each work item has one or more subtasks. All else being equal, work items with more subtasks ordinarily take more time to complete than those with fewer.

The modifications (independent variables) fall into four classifications:

  1. No modifications (control)
  2. Modification A alone, which prioritizes items with one or two subtasks over all other items
  3. Modification B alone, which presents the worker with work items of similar categories (and some other characteristics probably not relevant here) for a 30-minute period when possible
  4. Both modifications A and B at the same time

I want to measure productivity with three separate but correlated dependent variables:

  1. The number of work items completed.
  2. The number of subtasks completed.
  3. The total of a weighted score based on a category to which each work item belongs as applied to the same characteristics in the previous variable (a few thousand categories exist, and the category portions of the weights are predefined). More specifically, the score assigns the work item a fixed value plus a multiplier for the number of subtasks. I do not know how these values were determined.

I intend to apply the modifications at random to half-hour blocks of time. It is infeasible to apply the four classifications in equal proportions. The control needs to run most of the time (at least 75%). The remaining classifications can split the remaining time equally. Also, for practical reasons, it may or may not be possible to apply modification B alone.

I must apply modifications to all workers at the same time. That is, I cannot randomize workers into different workflow modifications and apply the modifications constantly.

The experiment can run for a maximum of two weeks, including thirteen consecutive hours on weekdays and nine consecutive hours on Saturdays, giving us a total of 296 half-hour blocks. The number of people completing work items at any given time will vary, but only one classification can be in place at a time, and that classification will apply to everyone working at that time. The number of work items suffices to keep everyone working the entire time they are there. The dependent variables will change based on the number of people working, so it may work better to express the dependent variables as a rate per person.

The classifications' effects, however, may not be truly independent. For example, I expect a drop in the number of work items completed (compared to the control's productivity without any modifications) for a not-yet-determined period of time after Modification A has ended and the control is reinstated. Also, I do not know the effect that applying both modifications at once will have compared to either modification individually. Logically, Modification B is more likely to have a greater effect when combined with Modification A than on its own, but I cannot be certain yet.

I have no useful baseline prior to starting the experiment, in part because these and other workflow modifications have been applied at will with only guesses (at best) as to their productivity improvements.

Given these facts:

  • Is the sample size large enough to provide meaningful results? If confidence makes sense for the analysis technique, I would likely consider $p > .9$ significant in this context. Minimum detectable effect would ideally be 5%. Assume that the number of work items completed per day is on the order of thousands to tens of thousands.
  • What techniques would be best to analyze these results? Some form of ANOVA seems like a possible starting point, but I could be completely wrong. Checking Wikipedia and other sites led me to no techniques that seemed perfectly fitting. Is a Chi-squared test sufficient to tell me if my process change was "good enough"? seems related, but I am unsure how the brief answer given in the comments would apply. @DemetriPananos suggests a Poisson regression. I initially rejected this approach because I was unsure that the distribution's assumptions held. Specifically, I am unsure that the events are truly independent, but I may be misunderstanding the concept.
  • Is this experiment design flawed, and if so, how might I fix it?

1 Answer 1


*Is the sample size large enough to provide meaningful results? I would likely consider 𝑝>.9 significant in this context.

This doesn't make sense to me. Can you justify this?

As for the sample size, it will depend on what method you use to analyze the data. If the outcome is "number of work items completed" this can be done with a Poisson regression. In order to give a proper sample size, I would need to know the smallest effect of the intervention you are willing to detect. This should be on the multiplicative scale (e.g. I want to detect an effect no smaller than 5% on productivity as compared to control).

What techniques would be best to analyze these results?

For the number of work items completed, Poisson regression seems preferable. I'd need to know more about this weighted score to recommend an approach. Can you say more about it? What is the score out of? Are the components of the score important at all? If you don't anticipate any of the scores to be near the boundaries (e.g. if the score is out of 100, then few scores should be near 0 or 100) then I think anova might be fine.

Is this experiment design flawed, and if so, how might I fix it?

Can you not randomize people to interventions instead of having them possibly perform all interventions? I think there might be a problem with correlation if you give every intervention to everybody. Ideally, I think it would be best to put 75% of people in control, and then randomize the rest to A, B, A&B.

  • $\begingroup$ I added details to the question to address these points. Minimum detectable effect of 5% seems reasonable. The weights are numbers of subtasks and a fixed value added to a multiplier of the number of subtasks. Unfortunately, I cannot randomize people in the way you suggest; besides being more difficult to implement in this situation, the workers would find their current productivity evaluations unfair, and modification A could never be applied across-the-board. $\endgroup$
    – Andrew
    Sep 7, 2019 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.