0
$\begingroup$

To paint a scenario, let's say that I'm interested in the impact of a new tool on the speed with which work is performed. I give 100 employees access to this new tool while another 100 use the current default tool. Over the course of a month, the treatment group and the control each complete approximately 10,000 tasks with their respective tool. The speed with which these tasks are completed is measured.

My question is whether an independent sample t-test is appropriate in this context, as the treatment is distributed at the level of the employee while the effect is measured at the level of the task. Given this discrepancy, there will be correlations within (but not across) the treatment and control groups at the level of the employee. I know this would deeply compromise ANOVA, but I am uncertain of the impact on a standard A/B test.

Does this compromise the results of a t-test?

If yes, what might a more appropriate research design be?

If I were to use the mean effect aggregating on employee, the n per group would drop from 10,000 to 100. Can anyone recommend suitable documentation in regards to power analysis when the unit of analysis is a mean?

$\endgroup$
  • $\begingroup$ Inter-class correlation means the correlation between two groups. I have never heard this term used. Intra-class correlation is the average correlation within groups. It seems like you mean neither one - you're trying to test for between group differences. Perhaps you should clarify the question. $\endgroup$ – Weiwen Ng Mar 1 '19 at 16:58
  • $\begingroup$ Thank you for the feedback. I will update the question. $\endgroup$ – StatStudent19 Mar 1 '19 at 18:18
0
$\begingroup$

You have repeated measures on two groups of subjects. Repeated measures ANOVA is one way to account for the fact that you are testing the same individuals twice, and it's a relatively simple extension of the ANOVA framework that you clearly already know.

More generally, you could also accomplish this using hierarchical linear regression, also known as a random effects model.

$\endgroup$
  • $\begingroup$ Thank you! I am certainly interested in the most appropriate test, and will look into repeated measures ANOVA (I have some familiarity with random effects models). At the same time, I am want to improve my understanding about why a conventional A/B testing framework would be inappropriate in this context. Is because the repeated measures causes an underestimation of the true variance, leading to an inflation of the false positive rate? $\endgroup$ – StatStudent19 Mar 1 '19 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.