0
$\begingroup$

While trying to validate the estimated impact by CausalImpact package I ran backtests on several dummy pre and post time-periods prior to the actual intervention but ended up with statistically significant impact results for several of the validation periods.

Actual test: Number of Predictor variables: 12 Pre-period time series length: 190 Post-period time series length: 50

I am certain that there were no interventions before the highlighted intervention. What could be causing the model to give statistically significant results in pre-intervention periods?

$\endgroup$

1 Answer 1

1
$\begingroup$

There are multiple explanations that might fit your scenario:

  1. Your model is correct and the null hypothesis is correct and the significant results are only happening by chance. This is less likely because you said several time periods are significant.
  2. You conducted multiple correlated tests in the pre-periods with out correcting your significance level therefore making it more likely to find significant results.
  3. You model is incorrect and the null hypothesis is true and because your test is under-representing the true variance in the data, it is making your test more likely to reject than it should.
  4. The null hypothesis is false in the pre-period because of unknown differences in test vs control or unknownn interventions.

Without your data and your code, it is nearly impossible to determine which situation your are in. My advice is to simulate data to ensure your understanding of the test is correct. If you are finding more than the expect number of significant results on simulated data, then your test statistic is invalid for your situation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.