How to detect a significant change in time series data due to a "policy" change? I hope this is the right place to post this, I considered posting it on skeptics, but I figure they'd just say the study was statistically wrong. I'm curious about the flip side of the question which is how to do it right.
On the website Quantified Self, the author posted the results of an experiment of some metric of output measured on himself over time and compared before and after abruptly stopping drinking coffee.  The results were evaluated subjectively and the author believed that he had evidence that there was a change in the time series and it was related to the change in the policy (drinking coffee)
What this reminds me of is models of the economy. We only have one economy (that we care about at the moment), so economists are often doing essentially n=1 experiments. The data is almost certainly autocorrelated over time because of this. The economists generally are watching, say the Fed, as it initiates a policy and trying to decide if the time series changed, potentially on account of the policy.
What is the appropriate test to determine if the time series has increase or decreased based on the data? How much data would I need? What tools exist?  My initial googling suggest Markov Switching Time Series Models, but not my googling skills are failing me at helping do anything with just the name of the technique.
 A: The Box-Tiao paper referred to by Jason was based on a known law change. The question here 
is how to detect the point in time. The answer is to use the Tsay procedure to detect Interventions be they Pulses, level Shifts , Seasonal Pulses and/or local time trends.
A: Josh said: 

josh: From the OP "What is the appropriate test to determine if the
  time series has increase or decreased based on the data? ". This I
  believe asks for a determination if the mean of the residuals has
  shifted not the parameters of some ARIMA Model. In my opinion you are
  recommending the wrong software/solution procedure but that's just my
  opinion. –  IrishStat Oct 28 '11 at 19:08

Suppose one starts with an AR(1) Model:
$$Y_t = \gamma + \phi*Y_{t-1} + E_t$$
Where ${E_t}$ is, say, a Gaussian Noise (mean zero and variance $\sigma^2$ 
The mean of this series.
The mean of the series is $\frac{\gamma}{1-phi}$
So, if for some time the parameters $\gamma$ and $\phi$ does not change, then so does the overall mean of the series. However, it any of these changes, necessarily the mean of the series will change. So, under piecewise stationarity, we are looking for changes of these parameters!
If structural models are assumed, Auto-PARM is the procedure to use. 
A: Looking through some old notes on structural breaks, I have these two cites:

Enders, "Applied Econometric Time Series", 2nd edition, ch. 5.

Enders discusses interventions, pulse functions, gradual change functions, transfer functions, etc. This article may also be helpful:

Box, G.E.P. and G. C. Tiao. 1975. “Intervention Analysis with Applications to Economic and Environmental Problems.” Journal of the American Statistical Association 70: 70-79.

A: Couldn't you just use a change point model, and then try to identify the change point using an MCMC algorithm such as Gibbs Sampling? 
This should be relatively simple to implement, provided you have some prior distributions for your data or the full conditional distirbution (for Gibbs).
You can find a quick overview here
A: If you were considering all time points as candidate change points (a.k.a. break points, a.k.a. structural change) then the strucchange package is a very good option.
It seem that in your particular scenario, there is only one candidate time point. In this case, several quick options come to mind:


*

*T-test: a t-test on the hours of concentration per day on the "before quitting" vs. "after quitting" periods. If you are concerned with day-to-day correlation, you could give up some observations so that you have long enough intervals to believe the days are no longer correlated. With this approach,you will be trading off power with simplicity. 

*AR: Fit an AR model with one dummy: "after quitting". If the predictor is significant, then you have a change. Using an AR, will capture the (possible) dependence between days. 

A: A few years ago I heard a talk by a grad student, Stacey Hancock, during a local ASA chapter meeting and it was on "structural break estimation" of time series. The talk was really interesting and I spoke with her afterwards and she was working with Richard Davis (of Brockwell-Davis), then at Colorado State University, now at Columbia. The talk was an extension of Davis et al. work in a 2006 JASA paper called Strutural Break Estimation for Nonstationary Time Series Models, which is freely available here. 
Davis has a software implementation of the method that he calls Auto-PARM, which he made into a Windows executable. If you contact him you may be able to get a copy. I have a copy, and here is example output on a 1,200 observation time series:
    ============== RESULTS ===============
  ISLAND           1
    SC=   1910.58314770669
    Breaking point/AR order
           1              1
         351              1
         612              3
    ======================================
 Total time:   5.812500

So the series is AR(1) in the beginning, at observation 351 the AR(1) process changes to another AR(1) process (you can get the parameters), and then at observation 612 the process changes to AR(3).
One interesting setting I tried Auto-PARM on was looking at weekly ATM withdrawal data that was part of the NN5 competition. I recall the algorithm finding structural breaks in late November of a given year, e.g. the beginning of the US holiday shopping season. 
So, how to use this algorithm via existing implementations? Well, again, you could reach out to Davis and see if you can get the Windows executable. When I was at Rogue Wave Software I worked with Davis to get Auto-PARM into the IMSL Numerical Libraries. The first language it was ported to was Fortran, where it is called Auto_PARM, and I suspect Rogue Wave will release a C port soon, with Python, C# and Java ports to follow.  
