# Determine if a person differs from typical performance

I have a database of workers, customers, and jobs, and I want to analyze my data to see if a given worker is performing well above or below the mean. I've come up with something of a solution, and I'm interested in hearing if there are any flaws in my current approach.

My goal is to see if a given worker has a lower than average score when it comes to converting first-time jobs into recurring jobs.

I do this by taking all first-time jobs that a given worker has been assigned to, and I then check to see if any subsequent jobs exist under the same customer. If there were subsequent jobs the worker gets a 1, otherwise they get a 0. This gives me something that looks like this:

Worker 1 [48 total records (first-time jobs)]
0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1,
--
Worker 2 [56 total records (first-time jobs)]
1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1,


I then take this data and calculate the mean. I do this by counting the total number of first-time jobs in my system (2,925), as well as the sum of the 1's and 0's. This gives me a mean of 0.38 (so 38% of all first-time jobs typically become recurring jobs). I then calculate the standard deviation, which in this case is 0.13.

I then look at each worker who has completed a minimum of 15 first-time jobs. This is so that I only analyze workers with a sufficiently large sample size, in order to increase confidence in the results.

I then come up with a mean score for each of these workers (the sum of the 1's and 0's, divided by the total number of first-time jobs). Finally, I convert this into a z score, which I do by subtracting the mean (for all the data) from the worker's mean score. I then divide the result by the standard deviation to obtain the individual z score. Finally, I put the results in a bar chart, which looks like this:

The dark bars are for any result with a z-score above 1 or below -1.

My questions are as follows:

1. Is this the correct approach, given my goals?
2. Does it make sense to limit this analysis to workers with a minimum of 15 records? The lower the threshold, the sooner I can perform this valuable analysis, but I want to make sure I'm using a large enough sample size to avoid problems.
3. If 15 is a good minimum, should I also use that as the maximum? For example, if Worker 1 has 15 records, and Worker 2 has 35 records, should I only analyze the last 15 of Worker 2's records? Or is it better to include all the data available?
4. Finally, what z-scores should I consider significant? Right now I am focusing on anything greater than 1, but is that too low?

I greatly appreciate any input. Thank you.

• How are jobs assigned to workers? That is, what if Worker 6 in your graph does slightly different work from the rest and ends up getting assigned customers who are much less likely to repeat? The old "all other things being equal... but are they actually equal?" – Wayne Sep 10 '12 at 20:54
• Yes, they are equal. I didn't include it in the above description, but I strip out any job types that are unlikely to recur, and only consider jobs that are equivalent. – Jeremy Sep 10 '12 at 20:56

1. Using a Z-score suggests that you know the standard deviation of the population in question. Thus, if your question is whether a given worker is above the workers currently employed by you, then yes, it is appropriate. If, however, you are interested in the population of all possible workers, of which the current workers in your study are only a sample, then Z is not appropriate. In this case you would want to use the t-statisitic.

2. and 3. It seems that both of these concerns could be handled by computing weighted averages. By this I mean use all of the scores but weight them by the number of observations for each employee.

3. What you should consider significant is highly idiosyncratic to a given situation. How will this information be used? What are the consequences of a Type I (false positive) or Type II (false negative) error? In basic behavioural research .05 is considered reasonable to accept for the possibility of a false positive (this is known as the alpha level). In certain medical research much lower alpha values are justifiably demanded. Once you've decided what is appropriate, you can find critical values in a table easily enough (e.g.) or get them from most statistical software like R very easily.

• Thanks for the response. I'm actually only doing this analysis on my workforce, so it's not a sample of a larger population. As far as #2 goes, how would I compute weighted averages in this case? Is there a formula available? – Jeremy Sep 10 '12 at 22:29
• This link explains the calculation of weighted averages. The weights in your case are the number of observations for each employee. This ensures that values from employees who have done many more jobs influence the average more and vice versa. – Marcus Morrisey Sep 12 '12 at 16:16