I have an experiment with two equally sized groups. These groups contain machines and the metric being measured is the failure rate of the of the groups.

In a typical experiment setting, the test group receives the treatment at the beginning of the experiment and we can use a sample size calculation to determine the length of the experiment.

However, in my case the treatment will be applied gradually and only to the worst performing machines in the test group for the duration of the experiment (which means in total only a small fraction of the entire group will receive the treatment). Essentially what is being measured here is whether this treatment plan being used only on the worst performers has an impact to the overall failure rate of the group.

One thing I'm confused about is how I should be adjusting the sample size calculation to account for this gradual change. Or perhaps I should be changing the design of the experiment?

For reference, I'm assuming the failures are Poisson distributed and I'm using the calculation below to estimate minimum sample size.

Sample Size Calculation


Partial Answer:

Well, you definitely have a confounding variable: machine performance. Let $M$ be machine performance, $T$ the treatment, and $Y$ the outcome. Then you have a causal diagram as follows:

enter image description here

The problem is, $T\leftarrow M\to Y$ is a backdoor path from $T$ to $Y,$ preventing you from getting at the true causal effect of $T$ on $Y.$ You can use the backdoor adjustment formula to fix the problem: $$P(Y=y|\operatorname{do}(T=t))=\sum_m P(Y=y|T=t,M=m)\,P(M=m).$$

| cite | improve this answer | |
  • $\begingroup$ Thank you so much for this, this is a great explanation and I didn't consider the fact that I'd have to adjust for the true effect. I realize now I have to make this adjustment to get the true effect but do you have a sense of how I should be approaching the experimental design (or sample size calculation) to determine the length of this experiment? I imagine that if I run the experiment with the computed days I have now and use the backdoor adjustment I'll still be missing the mark on the impact $\endgroup$ – ab1992 Jun 12 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.