2
$\begingroup$

I have a panel dataset of two groups X1 (control) and X2 (treatment), and my metric of interest Y (for both control and treatment). I have a dummy variable indicating what group they belong to, and a dummy variable indicating whether it is before or after intervention.

If I do a classic Difference in Difference analysis (Y ~ time+ treatment + time * treatment) it shows time (pre/post) as significant, but neither treatment or time * treatment are significant.

However, if I first calculate the difference between X2 and X1 (diff = X2-X1), and regress time against diff (diff ~ time) time is significant, and time has the exact same coefficient as time * treatment (since its the mean difference).

How should I interpret these results/differences in significance?

Do I interpret it as: while the absolute difference between the control and treatment between the two periods is significant, the impact of the difference between the two periods outweighs that of the difference between the groups?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

If you have measurements before and after some treatment, you have repeated measures. These are correlated to one another and thus violate the assumption of independent measurements. This means that neither of the models you have run produce valid standard errors.

Using a linear mixed effects model, you can account for the dependence between measurements of the same experimental units. For example, in R's lme4:

library(lme4)
LMM <- lmer(y ~ time * treatment + (1 | ID))
summary(LMM)

ID would then have to be a variable that indicates the experimental unit.

(Assuming the conditional distribution of your outcome is indeed normal.)

Apart from this, there is another difference in the first and second model you have run: The second model has a single error for every pair of $\{X_1, X_2\}$. This would only be valid if you could meaningfully construct such pairs (i.e. the treatments and controls are the same experimental units). Even if that were the case though, it would be more powerful to use a mixed model.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Can you elaborate more on what you mean by the same experimental units? My dataset is a panel dataset of two independant groups, of which one was exposed to a treatment and another was not. In my mind this seems like a typical use case for difference in difference using an OLS estimator? mailman.columbia.edu/research/population-health-methods/… $\endgroup$
    – sdhaus
    Commented May 1, 2020 at 1:56
  • $\begingroup$ In that case, an experimental unit is a single member of a single group $\endgroup$
    – Frans Rodenburg
    Commented May 1, 2020 at 1:58
  • $\begingroup$ These groups are not able to be separated, which is why I was thinking DiD would be useable (quote from the link): "DID is typically used to estimate the effect of a specific intervention or treatment (such as a passage of law, enactment of policy, or large-scale program implementation) by comparing the changes in outcomes over time between a population that is enrolled in a program (the intervention group) and a population that is not (the control group)." $\endgroup$
    – sdhaus
    Commented May 1, 2020 at 2:01
  • $\begingroup$ By the way, in the link in your comment, the author recommends using robust standard errors to account for autocorrelation, without explaining how to do so. A linear mixed model is a way to account for this autocorrelation. $\endgroup$
    – Frans Rodenburg
    Commented May 1, 2020 at 2:01
  • $\begingroup$ Right, just to clarify are you saying that: 1) the autocorrelation is likely causing the difference in significance between the analysis, and/or 2) both results are not usable and I either need to use a linear mixed model or OLS with clustered standard errors? $\endgroup$
    – sdhaus
    Commented May 1, 2020 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.