I have data for 72 employees, each of whom have between 10-30 evaluations from various supervisors. Each evaluation is anonymized, so we don't know who gave each evaluation, and we also can't be sure that multiple evaluations aren't written by the same person over different time frames.
If I group my employees into two bins that are roughly equal size (group A and group B) to run some kind of proportion test, and treat each evaluation as a new datapoint, am I violating the independence assumption?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ There are three forms of non-independence going on here: multiple assessments within the same subject, multiple assessments from the same supervisor (likely), a learning effect (presumably subjects getting negative or positive evaluations adapt to them somehow) and contamination (interaction between employees working in the same setting). $\endgroup$– AdamOCommented May 12, 2023 at 15:56
-
$\begingroup$ @AdamO oh wow so there are actually four then. Presuming I create a multi-level model with employee as the intermediate level to control for multiple assessments of the same subject (form 1), would it be okay to do an analysis as long as I acknowledge that the results may be flawed due to other uncontrollable sources of non-independence? $\endgroup$– Ryan FolksCommented May 12, 2023 at 18:01
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Yes, it is very likely. The evaluations for each employee are likely to be correlated, and if the same supervisor writes multiple evaluations, those are likely correlated. Each evaluation is in a cluster with the other evaluations for the same employee, and in a cluster with the other evaluations by the same supervisor. Note that we call measurements "clustered" when we think they are likely to be informative about other measurements that share a common characteristic. For example, if five different supervisors give an employee a very high rating, we can do better than a random guess on how supervisor 6 will rate an employee (all our evidence suggests that this employee should get a high rating). We know even more if we are aware that supervisor 6 tends to be either lenient or harsh with their evaluations.
A test that does not account for these clusters will likely underestimate the variance and thus is more likely to lead to an incorrect test conclusion. If the evaluations are anonymous though, there is not much you can do about it.
$\begingroup$
$\endgroup$
These data points are almost certainly not independent. It doesn't matter if there are many evaluations are written by the same person, as there are certainly many evaluations written about the same person. The dozens of evaluations of some particular employee are almost certainly not independent of one another, no matter who writes them. I would expect that an employee who has many good evaluations is more likely than usual to get another good evaluation - some employees consistently get good evaluations, while some consistently get poor evaluations. I would be very surprised if each employee had an equivalent random assortment of good and poor evaluations. There will be even greater dependence if the evaluation is also written by the same person, but as long as it's about the same person, it's not going to be independent.
answered May 12, 2023 at 15:12