To paint a scenario, let's say that I'm interested in the impact of a new tool on the speed with which work is performed. I give 100 employees access to this new tool while another 100 use the current default tool. Over the course of a month, the treatment group and the control each complete approximately 10,000 tasks with their respective tool. The speed with which these tasks are completed is measured.
My question is whether an independent sample t-test is appropriate in this context, as the treatment is distributed at the level of the employee while the effect is measured at the level of the task. Given this discrepancy, there will be correlations within (but not across) the treatment and control groups at the level of the employee. I know this would deeply compromise ANOVA, but I am uncertain of the impact on a standard A/B test.
Does this compromise the results of a t-test?
If yes, what might a more appropriate research design be?
If I were to use the mean effect aggregating on employee, the n per group would drop from 10,000 to 100. Can anyone recommend suitable documentation in regards to power analysis when the unit of analysis is a mean?