0
$\begingroup$

I am trying to analyse an experiment where in one region a treatment was given and in three other regions no treatment was given (control group). I have daily panel data for each region over a course of 12 weeks before the experiment and 2 weeks during the experiment. If I now run a Diff in Diff model and a two ways fixed effects model on this data, I get a significant positive effect when I use daily data, but no significant results with the weekly data (weekly aggregation is correct, I checked it multiple times).

Where could this difference come from? Shouldn't both give similar results? What is a general rule regarding the frequency of data for statistical significance?

Here a plot of the data with daily frequency: enter image description here

The vertical line indicates the start of the experiment in Georgia.

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome. When you say you ran a DD model and a two-way fixed effects model, does that mean you ran the classical model first, and then another DD model with unit and time fixed effects (generalized approach)? I’m curious if you’re getting different results between models, or is your question strictly focused on data aggregation up to the week level? $\endgroup$
    – Thomas Bilach
    Jul 15, 2020 at 15:47
  • $\begingroup$ Hi, I am focussing on the data aggregation (daily vs. weekly), not the comparison between both models. And yes, I run classical model and a DD (time and unit fixed effects). $\endgroup$
    – Blo4d
    Jul 16, 2020 at 7:22
  • $\begingroup$ A 4-unit panel is not a lot of data to begin with. When you aggregate to weekly from daily, you reduce your sample by a factor of 7. It is not surprising that significance goes out the window since significance is partly a function of sample size. $\endgroup$
    – dimitriy
    Dec 7, 2021 at 16:17

1 Answer 1

Reset to default
0
$\begingroup$

Your point estimates should be similar, but uncertainty will change.

Daily visitation across each state shows a lot of variation. Undulating patterns show more volatility, by day, as states move into the summer season. However, once you 'aggregate up' to the "week" level (or even higher), you will reduce outcome variation. As a result, your series will exhibit more stability across weeks. Aggregation also reduces the number of available observations, so you correctly note that estimates should remain unaffected. Your effects might be less precisely estimated, though. Juxtapose the standard errors associated with the daily versus weekly difference-in-differences estimates. I assume uncertainty will be inflated using the aggregated weekly series, which might dampen observed effects.

In my opinion, I would match the level of granularity observed in your plot.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.