How exactly can I determine if customer ratings are based on (employee) vs. just random variation? We have customer satisfaction surveys, and I can tell at least SOME portion of the variance is due to the employee that helped them. The surveys are all phrased -- how would you rate EMPLOYEE on meeting your needs? (paraphrasing a bit here.)
However, despite the phrasing, I have an inkling SOME portion of the variance is just due to the situation. The person in general might be a hot-head, have high (or low) expectations, having a good or bad day, or dealt with a crappy product/ software, etc.
I know what you're thinking --- given enough surveys (probably a ton) --- these extra factors, assuming they are randomly distributed given employees (which they generally are - times worked are generally the same) --- will balance each other as surveys approach --> large number.
Still, I'd like to know how valid, or how much variance, these may explain. I already know the people who select to do the survey are non-random, but comparisons should still be valid between employees.
For instance --- say I have 12 months of data for each employee. So 12 monthly scores for each (and also the raw surveys, there may be about 20-50 per month per employee).
Would analyzing simply the variance of these scores tell me much about their validity? Or the averages?
For instance, if one employee is scoring between 60-70% approval every month, and another is scoring between 80-90% each and every month --- that would indicate to me that the employee him/herself has a great deal of influence on the score.
However if someone averages 70%, and another person 80%, but both their scores have crazy swings between 40 and 100% every month, that would indicate to me, that I can't say for certain how much of that is due to the employee. This is just my initial biased estimation --- I know there are formal tests for this sort of thing, but I'm unsure if a specific formal test like a categorical ANOVA (?) is needed, or I'm making some pretty poor assumptions here. Any help much appreciated.
 A: This is precisely what an ANOVA-type analysis tells you: whether within-group (for one employee, in this case) variation is greater than variation across groups/among employees. A one-way ANOVA is a possibility if the equal variance assumption among groups is satisfied; it's non-parametric equivalent eg Kruskal-Wallis is an option otherwise. As you describe the problem, there is no particular advantage to using monthly averages vs. raw scores, since those scores are presumably independent surveys, and using all scores increases your power.
But you also seem to be interested in potential confounding factors: yet, based on your description, there does not appear to be data to attribute any of that (except month), unless you had data on the same customers evaluating several employees, a covariate of a customer's state of mind, or data on the type of product, etc. So your main assumption is that all other factors are random and unlikely to affect employee scores in any directional manner, and, with the limited information you provide, it seems somewhat justified (eg it is unlikely the same employee gets most of the customers who were having a bad day). 
Validity in this case is very much a construct of your question: when the test was designed, it was presumably justified that all these factors did not matter. Thus validity cannot be conclusively confirmed by observing or not observing a significant difference, e.g. no way of going back and checking if an employee's family stopped by every day to fill in their survey.
