Measuring performance with histogram

Question

We are measuring weekly staff performance, so that management can make informed decisions about task assignment. [1] To the end we have measured performance as follows:

• Workers are assigned batches of widgets to assemble and deliver
• Batches vary in widget quantity and complexity (complexity is not modelled, at least for now)
• Worker's performance is measured as number of widgets delivered per week
  • Delivery happens in batches, so if a batch of 10 is delivered this week, that counts as 10, even if 5 of the widgets were assembled last week
• Performance is visualised as a notched boxplot
• The boxplot is annotated with mean performance (total widgets / total weeks) -- outliers are included in this calculation
• Some workers have been working for 1 week and some for 6 months, so statistical significance varies per box (notch plots reveal this)

Assuming this approach is valid (challenges welcome), we are now faced with interpreting the following chart. We are looking to identify:

• fast, reliable workers: they can always be assigned large/complex batches
• workers who are "in their groove": manangement might try assigning them more complex batches
• slow and/or underperforming workers: need workload and/or complexity reduced, or need other forms of help or intervention

Can we use boxplot measures and the mean to identify these types?

To make the question more concrete, here are some sample interpretations for validation/correction (feel free to use or ignore):

1. Ben, Lee and Max have too few measures for statistical significance (weird notches), so take their interpretations lightly
2. Ian and Amy haven't delivered anything yet (we also know this because they've only been working a week)
3. Of the remaining workers, Ivy is the slowest, Ken is the fastest
4. Almost everyone suffers from too many weeks with no batches complete (Q1 = 0, 25% of weeks with no delivery)
5. Ben, Eli, Sia, Pat and Dom have a bigger delivery problem (Q2 = 0, 50% of weeks with no delivery)
6. Many workers are operating in their "zone" (Q2 < mean < Q3, so on average they are delivering in the high end of their IQR)
7. Max is a stable worker (small IQR), who could do better (mean < Q2), but we have to wait and see on this evaluation (weird notches implies low statistical significance)
8. Bob and Pat have roughly equal speed (B mean = P mean), but Bob is more likely to deliver faster (B Q2 > P Q2)
9. Ken is a bit faster Jen (K mean > J mean), but is far more likely to deliver faster (K Q2 much > J Q2)
10. Zoe is slower than Roz (Z mean < R mean), and less likely to deliver faster (Z Q2 < R 2)
11. Jen and Dom are both fast and in their groove, but Jen is much more likely to deliver (50% of Dom's weeks are 0 vs 25% of Jen's)

[1] We are aware of the perils of modelling performance on non-trivial, multivariate tasks, and the perils of managing only by the numbers. Our objective is only to make task assignment better-than-random and better-than-hunch.

Answer 1

If the batches differ in complexity then assigning complex batches to high performers is guaranteed to make them do worse. So unless you're accounting for the different difficulty of batches when comparing performance, the next time you go "oh, they're doing badly". People are terrible at appropriately adjusting for this via intuition (give them a measure and they'll use it).

The point was to measure performance, but if it's used the way you suggest it no longer measures performance, it confounds it with task complexity. If there's some good measure of task complexity/difficulty, it would be possible (e.g. via regression methods or similar) to account for this and still have a measure of performance (by giving a way to have a complexity-adjusted performance).

---

Response to the discussion of the boxplots (using your numbering):

\1. The notches aren't "weird"; the interval they give is just further out that at least one of the quartiles (/hinges). That might just by accident be too small a number for you to regard it as reliable, but if you apply some sensible criterion to decide your n is large enough to use, it probably won't really correspond to this -- maybe it would be half as many or twice as many as saying "the plot looks funny".

\3. On sample mean, sure. But not on all sensible measures. Note also that these estimates are so noisy that every median (and every mean, though that's not really relevant) is inside Ken's notch-interval, so saying "Ken is fastest" is like saying my coin is better than your because when we each tossed 10 times I got 6 heads and you got 4. It's probably just noise.

\4. Nothing about the plot makes 25% "too many", though the interpretation that there are at least 25% zeros is the case for everyone but Max.

\6. This is simply a result of the fact that the distribution is reasonably skew. "In their zone" doesn't mean anything I can discern but whatever it's intended to mean, this isn't evidence for anything than a count variable with a small mean (especially with a fair chance of a zero) is likely to be pretty right skew. There's no basis for assigning much meaning to that, it's almost certainly a consequence of the thing you're measuring, and the small sample sizes would suggest that anything above "it's just the nature of the process" is probably noise.

\8. -- 11. No, it's probably just noise. You don't have nearly enough data here for this level of detailed conclusion. "In their groove" doesn't mean anything to me, but whatever it is, this doesn't show it