I'm attempting a 'difference-in-differences' analysis of a health policy intervention.

Scenario: Health clinics are paid for the percentage of eligible patients who they give the right treatment to. The clinics are measured/paid separately for different treatment indicators (e.g. treatment for diabetes, for high blood pressure).

One year, they stopped payments for some of the indicators (but others continued). I want to measure if stopping the payment had an impact on performance (relative to the control group).

I have performance data for every single health clinic in the country - for the intervention and control group (both pre- and after- intervention.

My questions are:

  1. If I have data for every single health clinic in the country (it's not a sample), should I still calculate the Standard Error and confidence intervals for the difference-in-differences (DiD) estimator?

  2. How best should I calculate the SE and CI for the DiD estimator? Should I use just the mean average, or can I factor in the individual pairs of measurements (for each clinic)? There is a pre- and post- measurement, for the control and the intervention, for every single clinic.

  3. Each clinic has varying numbers of eligible patients (I have the numerator/denominator for each); can I factor that into the estimation of the impact?

I'm doing the analysis in STATA.

  1. Yes. We often conceptualize uncertainty as making inferences about the population from a sample but it is far from the only appropriate use of hypothesis testing. Think of each clinic's outcome as the realization of a random variable - your question is not whether the averages for the two groups of clinics diverged after the policy change, which you can calculate exactly, but whether the divergence is more than would be expected due to the underlying randomness.

  2. The easiest way to implement DiD is to formulate it as a regression and just run OLS: $Performance_{c,t} = \alpha + \beta_{1}*D_{after,c,t} + \beta_{2}*D_{intervention,c,t} + \beta_{3}*D_{after,c,t}*D_{intervention,c,t} +\epsilon_{c,t}$
    where $D_{after,c,t}$ is a dummy variable that is set to 1 for observations after the change and 0 before it and $D_{intervention,c,t}$ is a dummy variable that is set to 1 for observations in the intervention group.
    $\beta_{3}$ is your DiD estimator and the regression will give you it's standard error, confidence interval, and t-statistic.
    This also makes it easy to include additional explanatory variables or adjust the standard errors for clustering if you think it makes sense in your setting.

  3. You can use that information, but it is no longer a "difference-in-differences" analysis. It is often called a "dose-response" analysis. For an example, see this paper: Health insurance and opioid deaths: Evidence from the Affordable Care Act young adult provision.

| cite | improve this answer | |
  • $\begingroup$ Thanks very helpful. So my understanding is that the DiD estimator will be ((InterventionAfter-ControlAfter)-(InterventionBefore-ControlBefore)). So say Beta3 (DiD effect) is -4.60%. What formula do I use to calculate the t-value? (and the SE)? And presumably, if the 95% interval crosses 0, then it may be the intervention had no significant effect? Thanks. $\endgroup$ – hassapikos Apr 10 '19 at 14:11
  • $\begingroup$ Yes, ((InterventionAfter-ControlAfter)-(InterventionBefore-ControlBefore)) is one way to calculate the DiD estimator, but you should get the same number if you run the linear regression I specified with the standard OLS estimator of $(X'X)^{-1}X'y$. Any statistical software will easily give you the coefficient estimates and their t-stats and confidence intervals, there is no need to calculate them manually. I'm not going to write the formulas here, they get messy for multivariate regression. $\endgroup$ – Matt P Apr 10 '19 at 15:20
  • $\begingroup$ Great thanks for your help. $\endgroup$ – hassapikos Apr 10 '19 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.