I'm working with a panel data from a quasi-experimental study with the following characteristics

1) Unbalanced covariates between the control and the treatment group (according to Hotelling's T-squared test), though I've read that it's not cause for concern (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4310023/);

2) High level of attrition, though similar in rate for both the control and treatment group, and no statistical difference in covariates between drop-outs from the two groups (Pearson chi2 for each covariate) but significant using Hotelling's T-squared test;

3) Treatment assignment at the regional level, participants were randomly selected in three random towns within a region;

4) Defiers in both the control and the treatment group;

5) Most likely heterogenous effects by region.

I was thinking of using a DID analysis, given that the control and treatment groups' covariates and key indicators differ, and then use a LATE approach to estimate the average treatment on the treated. I was wondering if:

a) that would be the correct approach? b) I read somewhere that DID is ATE, and that for LATE, I would need to estimate the ITT. How can I estimate the ITT? c) can I estimate ATT on DID? d) any suggestions on different approaches, given the type of data (i.e. FE, PSM)?


If you are assigning a treatment, it is not a "quasi-experimental" study as you say in the header.

1) The paper which you cite has a subtle claim: statistical testing of differences in baseline covariates is bad practice. Historically, RCT analyses were initiated with a litany of t-tests and chi-square tests, and provided none of these were significant, the authors claimed that "randomization was good" or "balanced". This is just untrue and misleading, especially with deliberate variable omission which occurs frequently in practice. Regrettably, it's true that large imbalances in baseline covariates between randomization arms will lead to biased or inefficient analyses. The solution they propose is simply to adjust (or perhaps block-randomize) for strongly prognostic variables.

2) There is no widely accepted solution to this problem. The approach most often used is what we'd call an intent-to-treat analysis. Use the data from the non-compliant participants up until the time they leave the study (either with censoring or including them as imbalanced clusters in a mixed model). Some sensitivity analyses can be done to guess at what their trajectory would have been had they stayed in the study. Last-observation-carried-forward (LOCF) is never recommended. You can use multiple imputation via chained equations if there is no growth component (time-covariate interactions) in the model, or you can use some Martingale processes like a simple random walk with Markov Chain to forecast outcomes. Another type of sensitivity analysis is worst-observation-carried-forward (WOCF) where you use the worst observed or unobserved outcome (within an individual or the whole panel) in those who drop out. Presumably drop-out is higher in the treatment group because the treatment is more invasive and different from standard of care, so this can be seen as a conservative approach; however you must verify this.

3) I will just say that you should read this: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4458010/. Community randomized studies are important and widely popular for evaluating policies and interventions.

4) Again, for any evaluation, analyze "defiers" with intent-to-treat. If I randomize a set of cancer patients to chemotherapy, but 50% of them tolerate the treatment so poorly they refuse to show up for infusions, I can't say the treatment was effective in 100% of the 50% who showed up, it was only 50% effective.

5) With community randomized studies it can be difficult with perfectly nested effects due to site. The best way to handle this variability per the above referenced article is to adjust for important site based predictors of the outcome, where available. This increases the generalizability of the results. While adjusting for site directly is tempting and would normally be done in other circumstances, with few sites (typically the case) perfectly nested within treatment, there can be some forms of bias that occur.

  • $\begingroup$ Hi AdamO, thank you for your answer! Extremely helpful! I had a follow-up question - how would you test ITT if the covariates differ? From my understanding, ITT can be done with a t-test between two statistically similar groups. Because they differed from the baseline (though not drastically), I thought it would be best to perform an initial DID analysis (but DID=ATT). From then, how would I implement a LATE on the compliers (egap.org/methods-guides/…) on a DID? I'll then analyse the data as per your recommendations! Thank you! $\endgroup$ – Nick Sep 16 '17 at 4:43
  • $\begingroup$ @Nick ITT is not a test, it is referred to as an analysis (more of an approach, really). To say you did an ITT analysis tells us who is included in the sample and how their randomization/treatment (assignment) is coded. It has no bearing on the type of model used. Choose the analysis (per adjustments, coding of the outcome, and type of inference) that suits the problem. continued... $\endgroup$ – AdamO Sep 16 '17 at 14:14
  • $\begingroup$ @Nick I'm glad you cited this LATE thing... I could not google a source. The converse of ITT which I have seen in clinical trials is a per protocol analysis: here the coding of the main effect "treatment assignment is restricted to treatment. As your paper suggests, never-takers are excluded from analyses. They never conform to what is considered a "safety population" (a group who was exposed to investigational treatment and comparable control). Also implied is that, in longitudinal studies, participants who take intermittently contribute imbalanced cluster data. $\endgroup$ – AdamO Sep 16 '17 at 14:18
  • $\begingroup$ Hi AdamO, thanks for your comments. I didn't fully understand your first point - if I were to use LATE (or per-protocol analysis, as from what I read, they are similar), I would first need to derive the ITT, as LATE is evaluated by dividing ITT by the compliers rate. I know how this could be done when the control and treatment groups have balanced covariates. However, if this is not the case, how could I estimate the ITT? Thanks again! $\endgroup$ – Nick Sep 17 '17 at 2:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.