I have a time series which represents call attempts to a cellular network. The data was recorded over 7 days using hourly intervals. I plan to make a brief change (for only a few hours) to the network which will either increase or decrease the call attempt value during that period of change.

How can I measure with confidence whether my brief change had any impact using statistics? I've read about change point analysis here but knowing my data, the changes in my data could be completely random due to the time of the day or day of the week etc and it could highlight multiple change points that mean nothing.

I'm afraid that my change will be masked by temporal events in the data and I will not be certain if my change had any impact.

Below is a sample of what a single cell variable looks like over 7 days. If you have any suggestions please shout.

enter image description here

  • 2
    $\begingroup$ check out the CausalImpact R package: google.github.io/CausalImpact/CausalImpact.html, with an introduction video here: youtube.com/watch?v=GTgZfCltMm8. I don't like all the fake or perfect data circumstances in their examples, so here's something I did with it: markhw.com/blog/viceroy-mac-demarco $\endgroup$
    – Mark White
    Apr 16 '18 at 23:12
  • $\begingroup$ @MarkWhite, thanks for the reply. Rather than just throw my data into a black box I am reading the original paper to get a better intuition about the workings of the model. My data above is from a cell site in say Florida where my intervention was solely focused, I have similar time series from all over the country, can I use the same variable (call attempts) from California, Boston, Chicago...... as the predictor variables? $\endgroup$
    – TheGoat
    Apr 17 '18 at 23:05
  • $\begingroup$ yeah, that seems reasonable to me. You’re basically using all the covariates to build a model that will estimate what things would have been like IF you didn’t do the treatment. Then it just compares what you DID observe to this estimated counterfactual. I would look up the potential outcomes framework—sometimes called the Rubin causal model in more technical papers—for a theoretical background on that idea of counterfactuals. $\endgroup$
    – Mark White
    Apr 18 '18 at 4:36

You are quite right in your reflection about "temporal effects" . You can safely ignore any advice that doesn't explicitly treat these effects while attempting to identify and measure latent activity. Your data can be used to identify break points either in the expected value (intervention detection) or changes in variance . See http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html which covers these first 2 issues. Another issue is break-points in parameters over time which can be resolved/detected by using Chow's procedure https://en.wikipedia.org/wiki/Chow_test sequentially . All three of these considerations have been routinely implemented in AUTOBOX which I helped to develop.

This topic is sometimes referred to as the "interrupted experiment" in the social sciences . Early work by Box and others focused on the case where the change point was known a priori. Developments in the late 70's by I. Chang and G. Tiao spoke to the issue of finding the change point(s).

If you wish to post your data I will try and help further.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.