# How exactly can I determine if customer ratings are based on (employee) vs. just random variation?

We have customer satisfaction surveys, and I can tell at least SOME portion of the variance is due to the employee that helped them. The surveys are all phrased -- how would you rate EMPLOYEE on meeting your needs? (paraphrasing a bit here.)

However, despite the phrasing, I have an inkling SOME portion of the variance is just due to the situation. The person in general might be a hot-head, have high (or low) expectations, having a good or bad day, or dealt with a crappy product/ software, etc.

I know what you're thinking --- given enough surveys (probably a ton) --- these extra factors, assuming they are randomly distributed given employees (which they generally are - times worked are generally the same) --- will balance each other as surveys approach --> large number.

Still, I'd like to know how valid, or how much variance, these may explain. I already know the people who select to do the survey are non-random, but comparisons should still be valid between employees.

For instance --- say I have 12 months of data for each employee. So 12 monthly scores for each (and also the raw surveys, there may be about 20-50 per month per employee).

Would analyzing simply the variance of these scores tell me much about their validity? Or the averages?

For instance, if one employee is scoring between 60-70% approval every month, and another is scoring between 80-90% each and every month --- that would indicate to me that the employee him/herself has a great deal of influence on the score.

However if someone averages 70%, and another person 80%, but both their scores have crazy swings between 40 and 100% every month, that would indicate to me, that I can't say for certain how much of that is due to the employee. This is just my initial biased estimation --- I know there are formal tests for this sort of thing, but I'm unsure if a specific formal test like a categorical ANOVA (?) is needed, or I'm making some pretty poor assumptions here. Any help much appreciated.

• Sounds like you have a pretty good handle on your problem. Is it just one dimensional? i.e. how many questions do customers answer? Be careful about attributing variance to unobserved variables. What if an employee really does swing between 40 and 100 because of low blood sugar, or they're just generally more volatile, or some other behavioural influence? – Neil Dec 15 '14 at 22:53