# Proper statistical procedure for repeated RCTs on the same group (without replacement)

Setting:

• Longitudinal data on outcome Yi,t of a group of individuals, i={1,...,N}, over time, t={1,...,T}
• On this group, a sequence of RCTs (r={1,...,R}) staggered over time is applied.
• Each RCT is measuring the effect of a specific treatment and treats a small percentage of the population.
• Once an individual is treated in a certain round (r=x) then he is no longer eligible for other future RCTs (rounds).

I am trying to measure the specific impact of each treatment (eq 1), as well as the average impact of the treatments (eq 2).

1. Yi,t,r = ALPHArTREATEDi,r + BETAr(TREATEDi,r*I_POST_TREATMENTt,r)
2. Yi,t = ALPHA*TREATEDi + BETA *(TREATEDi *I_POST_TREATMENTt)

The control group of that same round (r=x) on the other hand may be "contaminated" by being treated in a future RCT. Because trials treat a small percentage of the population each round (without replacement), and there aren't that many rounds, a significant proportion of the control group remains untreated over the entire period.

What is the correct statistical procedure to adopt here?

Is there a technical term for this sort of setting?

Any relevant literature I should look at?

EDIT: To try to collect me inputs, I cross posted on Stata List here

• Your problem sounds like it has some parallels to wait-list control designs that are often employed when performing intervention research. Though these designs may not completely align with your problem, there could be some useful approaches that inform your decisions. – Matt Barstead Jan 24 '18 at 16:28

As I understand it, your issue here is that you are worried about "contamination" of the control group(s), due to the fact that multiple RCT interventions are being applied in your analysis. If you analyse each RCT intervention separately, then you will indeed get this contamination, but if you perform a multivariate regression, allowing for all the RCT interventions in the model at the same time, this should not be a problem.

Since none of the RCT interventions are applied to the same person, it ought to be relatively simple to measure their effects over time without confounding (i.e., there are no cross-effects to consider). For each individual $i$, define the values $R(i) = 1, ..., R$ and $T(i) = 1, ..., T$ as the respective treatment and treatment time for that individual.

We could model the data to allow an effect for each of the $R$ interventions, and allow the effect to vary over time according to the delay that has occurred since intervention. (This is a generalisation of your suggestion to have a treatment effect and a single post-treatment effect. It would allow for a gradual drop-off of each effect over time.) To do this, we could model the data as:

$$Y_{i,t} = \beta_0 + \sum_{r=1}^R \sum_{d=0}^{T-t} \beta_{r,d} \cdot \mathbb{I}(r = R(i)) \mathbb{I}(t=T(i)+d) + \sigma \epsilon_{i,t},$$

where $\epsilon_{i,t} \sim \text{IID N} (0, 1)$ are error terms. This regression model would allow you to use all the interventions in a single model, so that each intervention is accounted jointly, and any resultant "contamination" of the control group is accounted for in the model. If you have other explanatory variables (e.g., subject characteristics), you can add them in also. (Note that identifiability requires that we only use parameters $\beta_{r,d}$ that correspond to an observable delay for the treatment under consideration. If you are lacking some treatment/delay combinations you would take these parameters out of the model.)

If you were to use a regression model like this, fitting the model would give you estimates of each treatment effect at each allowable delay value (with $d=0$ being the initial treatment effect and $d>0$ being the subsequent post-treatment effects). Since all effects are estimated simultaneously, and there are no instances of exposing an individual to multiple interventions, there should be no problem with the fact that you have multiple interventions.

Caveat: Obviously this is just a base model, using multiple linear regression. I have no idea if this would fit your data well. You might want to alter the model to a GLM, or add some random effects terms, or do something else to make it fit better. (Personally I would add mixed-effect terms.) The purpose here is just to show that you can formulate a model that simultaneously measures all treatment effects.

• This is great, Tks. I intend to add period and person fixed effects. What about a statistical term for this setting, so pople can easily recognize it? – LucasMation Jan 29 '18 at 11:03
• You could still do this within regression; it would just be a matter of adding those two factor variables into the model. They should be easy for people to recognize. – Ben - Reinstate Monica Jan 29 '18 at 21:19

Another option for this analysis is synthetic cohort methods, which will relax the parallel trends assumption of difference-in-differences.

The multiple treatments at different times version of SC is detailed here:

Cavallo, E., Galiani, S., Noy, I., and Pantano, J. 2013. Catastrophic natural disasters and economic growth. Review of Economics and Statistics, 95(5):1549–1561, Dec 2013.

There is a user-written Stata package called synth_runner that implements this. See Example 3 in the help file. This repo contains the relevant papers in pdf format, including the one I cited above.