4
$\begingroup$

I have panel data of sales for many stores in two comparable cities. One of the cities holds a special event once a month (the treatment) which is expected to boost sales across the board on that day. I would like to estimate the average treatment effect on the treated, i.e. identify whether or not the special event does what it should.

While the two cities are quite comparable (and I can find a number of controls for the stores), sampling is clearly not random. Furthermore, there is autocorrelation in sales between dates at each individual store, and it seems reasonable to think the monthly seasonality of the treatment has some impact.

My question is twofold.

1/ Do you think running a difference-in-differences analysis in this “experimental” setup makes sense at all?
2/ The examples of diff-in-diffs I have seen are all pre-treatment/post-treatment. Do you have any idea of how I could account for the periodicity of the treatment? It seems wrong to consider each month separately because of the autocorrelation.

Thank you very much for your help!

Improve this question
$\endgroup$
6
  • $\begingroup$ Are you wanting to do an analysis of number of sales (i.e count data) or value of sales? I'm thinking of a repeated measures design, but it would depend on the variables you want to use, and your dependent variable. $\endgroup$
    – Michelle
    Jan 21, 2012 at 1:11
  • $\begingroup$ I was mostly thinking of using the aggregate sales number for each store ($ value), but I could also use the number of individual customers. On the right-hand side, I was considering putting the date, a boolean for the treatment date, and possibly some controls for individual store characteristics. I am not sure if repeated measures design is appropriate: I would not be using my “control” city, and sample size is not really an issue as I have many stores and several years of data. Also, every store in the treated group is treated on the same day every month. $\endgroup$
    – René R
    Jan 21, 2012 at 6:50
  • $\begingroup$ Sorry to ask another question, when you say you were thinking of aggregate sales data, do you mean aggregate by day or aggregate by a longer period? If you had multiple days of sales data by store, that is a repeated measures design, which is completely "do-able". What software do you have available for analysis? $\endgroup$
    – Michelle
    Jan 21, 2012 at 9:11
  • $\begingroup$ I did mean aggregate by day. Repeated measures design is doable, but it means abandoning the data for the untreated city, hence not being able to control for other underlying market conditions and seasonality of sales which would impact simultaneously both cities. My preferred software is R, but I can use something else if needed. $\endgroup$
    – René R
    Jan 21, 2012 at 9:22
  • $\begingroup$ I thought you would use the treated/untreated as a dummy group variable. For seasonality, is there a series of dummies that can be used, e.g. a dummy for weekend sales? Is your data matched day-for-day at all sites? I was wondering about time series analysis, but couldn't work out how to handle months of different lengths to account for the sales day seasonality. $\endgroup$
    – Michelle
    Jan 21, 2012 at 9:45

2 Answers 2

Reset to default
1
$\begingroup$

Let $y_{ct}$ be the outcome in city c at time t, $x_{ct}$ be a city-specific variable at time t, $z_t$ be a non-city-specific variable at time t (e.g., day of the week), and $T_{ct}$ be the treatment indicator (e.g., boolean). For cities A and B, $$\begin{align*}y_{at} &= \alpha_a + \beta_a x_{ct} + \gamma_a z_t + \delta T_{at} + \epsilon_{at} \\ y_{bt} &= \alpha_b + \beta_a x_{bt} + \gamma_a z_t + \delta T_{bt} + \epsilon_{bt} \end{align*}$$ Assume that city A was treated, while city B was not. Let's take the difference: $$\begin{equation*} y_{at} - y_{bt} = (\alpha_a - \alpha_b) + \beta_a x_{at} + (- \beta_b) x_{bt} + (\gamma_a - \gamma_b) z_t + \delta T_{at} + (\epsilon_{at} - \epsilon_{bt}) \end{equation*}$$ What's the magic of difference-in-differences?

  • Suppose that $x$ is unobservable. If $\beta_a x_{at} = \beta_b x_{bt}$, for example, if the cities have the same value of the covariate and the same impact of that covariate, then this unobserved factor cancels out.
  • Suppose that $z$ is unobservable. If $\gamma_a = \gamma_b$, this term drops out.

These features are why we are interested in finding comparable cities---unobservable components are assumed to be the same across cities and they cancel out.

(Note: this is a single difference. Where's the other difference? We care about $\delta$, which is the difference between non-treatment and treatment periods.)

Now, we are left to figure out what to do with the error terms. If we think that $y$ has a unit root, we should use $\Delta y_{ct}$ rather than $y_{ct}$ from the beginning. Do a Dickey-Fuller or KPSS test on each city's $y$ series to find out.

Otherwise, we should refer to this paper: How Much Should We Trust Differences-in-Differences Estimates? by Bertrand, Duflo, and Mullainathan. Collapsing your data into a before-and-after way actually helps to alleviate the autocorrelation, the opposite of your intuition. Better ways involve the bootstrap or clustered standard errors.

Improve this answer
$\endgroup$
0
1
$\begingroup$

Caveat 1: I have just starting getting into mixed effects models, so I am a little worried I am over-finding applications for this general type of method. I'm familiar with the nlme and lme4 packages, and there could be a better package than those that I am unfamiliar with. Caveat 2: I don't have much call in my work for fitting nested models, so I welcome a more experienced person to check my suggestion below and provide feedback as I can learn from that as well.

I was thinking along the lines of trying a linear mixed methods approach. You'll need to check the normality assumptions for your sales data, but these methods mean you could fit store as a random effect, which would take account of the repeated measures aspect of your data.

As you have a nested design, perhaps a model like this might work:

DailySales$ ~ [store size] + [weekend/holiday indicator] + [city indicator] + [treatment indicator] + (1|[city indicator]/[store ID])

This model suggests fitting store as a random effect grouped within city (the last term).

Have a look at the lme4 package in R, and at the lmer option. I'm not sure if all the stores have the same number of opening hours, and if some stores are open longer than others, that may be another variable to fit.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot, I'm looking into this class of models and the lme4 package as I'm unfamiliar with them, and I'll report my findings. $\endgroup$
    – René R
    Jan 21, 2012 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.