I hope this is the right place to post this, I considered posting it on skeptics, but I figure they'd just say the study was statistically wrong. I'm curious about the flip side of the question which is how to do it right.
On the website Quantified Self, the author posted the results of an experiment of some metric of output measured on himself over time and compared before and after abruptly stopping drinking coffee. The results were evaluated subjectively and the author believed that he had evidence that there was a change in the time series and it was related to the change in the policy (drinking coffee)
What this reminds me of is models of the economy. We only have one economy (that we care about at the moment), so economists are often doing essentially n=1 experiments. The data is almost certainly autocorrelated over time because of this. The economists generally are watching, say the Fed, as it initiates a policy and trying to decide if the time series changed, potentially on account of the policy.
What is the appropriate test to determine if the time series has increase or decreased based on the data? How much data would I need? What tools exist? My initial googling suggest Markov Switching Time Series Models, but not my googling skills are failing me at helping do anything with just the name of the technique.