We have customer satisfaction surveys, and I can tell at least SOME portion of the variance is due to the employee that helped them. The surveys are all phrased -- how would you rate EMPLOYEE on meeting your needs? (paraphrasing a bit here.)
However, despite the phrasing, I have an inkling SOME portion of the variance is just due to the situation. The person in general might be a hot-head, have high (or low) expectations, having a good or bad day, or dealt with a crappy product/ software, etc.
I know what you're thinking --- given enough surveys (probably a ton) --- these extra factors, assuming they are randomly distributed given employees (which they generally are - times worked are generally the same) --- will balance each other as surveys approach --> large number.
Still, I'd like to know how valid, or how much variance, these may explain. I already know the people who select to do the survey are non-random, but comparisons should still be valid between employees.
For instance --- say I have 12 months of data for each employee. So 12 monthly scores for each (and also the raw surveys, there may be about 20-50 per month per employee).
Would analyzing simply the variance of these scores tell me much about their validity? Or the averages?
For instance, if one employee is scoring between 60-70% approval every month, and another is scoring between 80-90% each and every month --- that would indicate to me that the employee him/herself has a great deal of influence on the score.
However if someone averages 70%, and another person 80%, but both their scores have crazy swings between 40 and 100% every month, that would indicate to me, that I can't say for certain how much of that is due to the employee. This is just my initial biased estimation --- I know there are formal tests for this sort of thing, but I'm unsure if a specific formal test like a categorical ANOVA (?) is needed, or I'm making some pretty poor assumptions here. Any help much appreciated.