CausalImpact Conflicting Results - Posterior prob. v Confidence Interval

Question

I'm getting conflicting results from a Causal Impact analysis I'm running. The 95% C.I. indicates that 0 is included and thus the impact is not significant; however the Posterior Probability is <.05, so the impact is significant. Results are below:

                         Average        Cumulative  
Actual                   107            1284        
Prediction (s.d.)        99 (5)         1183 (60)   
95% CI                   [89, 109]      [1068, 1304]

Absolute effect (s.d.)   8.4 (5)        100.9 (60)  
95% CI                   [-1.7, 18]     [-19.8, 216]

Relative effect (s.d.)   8.5% (5.1%)    8.5% (5.1%) 
95% CI                   [-1.7%, 18%]   [-1.7%, 18%]

Posterior tail-area probability p:   0.04343
Posterior prob. of a causal effect:  95.657%

Any ideas how I should interpret this?

Answer 1

You can get a nice summary with summary(put_model_object_name_here, "report") which might aid interpretation.

This is a two-sided versus one-sided hypothesis test issue. The posterior probability that the intervention had any effect is expressed in terms of a one-sided p-value. Checking whether the posterior interval for the effect includes zero corresponds to a two-sided hypothesis test. Usually these agree, but here you have found an edge case where the choice makes a substantive difference.

I would not use a 5% cutoff so rigidly here, since you might be able to reject the null of no effect at 6 or 7%. I would also advocate using info about the setting and what you hope to learn/do in deciding whether a two-sided or a one-sided test is more appropriate. The IDRE link above goes into this essentially non-statistical decision.