Sample Size Calculation considering Gradual Improvements to Treatment Group

Question

I have an experiment with two equally sized groups. These groups contain machines and the metric being measured is the failure rate of the of the groups.

In a typical experiment setting, the test group receives the treatment at the beginning of the experiment and we can use a sample size calculation to determine the length of the experiment.

However, in my case the treatment will be applied gradually and only to the worst performing machines in the test group for the duration of the experiment (which means in total only a small fraction of the entire group will receive the treatment). Essentially what is being measured here is whether this treatment plan being used only on the worst performers has an impact to the overall failure rate of the group.

One thing I'm confused about is how I should be adjusting the sample size calculation to account for this gradual change. Or perhaps I should be changing the design of the experiment?

For reference, I'm assuming the failures are Poisson distributed and I'm using the calculation below to estimate minimum sample size.

Answer 1

Partial Answer:

Well, you definitely have a confounding variable: machine performance. Let $M$ be machine performance, $T$ the treatment, and $Y$ the outcome. Then you have a causal diagram as follows:

The problem is, $T\leftarrow M\to Y$ is a backdoor path from $T$ to $Y,$ preventing you from getting at the true causal effect of $T$ on $Y.$ You can use the backdoor adjustment formula to fix the problem: $$P(Y=y|\operatorname{do}(T=t))=\sum_m P(Y=y|T=t,M=m)\,P(M=m).$$