# Statistical significance in time series and sub-series

So, when I did first year stats in undergrad, we did an experiment where we tampered with a bunch of coins, to see if it would cause a statistical difference in the results. This is a graph of the ratio of $heads:tosses$ for each series of flips:

We took the full data set in each case, and found that there were no significant differences from a null hypothesis of 50% heads.

Had we stopped at 100 flips, or 150 flips we would have probably concluded that the Cupped coin was significantly biased. Would this have been invalid, and why? In particular, does it mean anything that the ratio is outside the 95% confidence interval more than 5% of the time?

-
The reason I bring this up years after completing the assignment is that there has been much discussion of the Global Warming "Hiatus" of the last decade or so (and also some clear explanations about how the heat is still there, it's just entering the oceans rather than the atmosphere). To me, the "Hiatus" pattern (a drop down to the lower 95% confidence interval bound of the prediction) appears qualitatively similar to me to the period around 100 flips in the image above. – naught101 Apr 16 '14 at 4:06

Had we stopped at 100 flips, or 150 flips we would have probably concluded that the Cupped coin was significantly biased. Would this have been invalid, and why?

That depends. I'm going to give you a series of reasons why you maybe shouldn't find it surprising at all.

If your decision to stop then had been entirely a priori, that one would be significant at the 5% level.

If the decision to stop then was based on observing its significance, then yes it's invalid, at least in the sense that individual tests are no longer a 5% test as claimed. You'd actually in the area of sequential testing for which the intervals are different.

http://en.wikipedia.org/wiki/Sequential_analysis

http://en.wikipedia.org/wiki/Sequential_probability_ratio_test

Leaving that aside, you don't state what your 95% and 65% intervals actually are. I assume they're the intervals for an individual proportion, and as such you expect on average that about 5% of the values you generate will lie outside the bounds.

But keep three things in mind when considering that 5%:

1) you're running 6 coins. If nothing is going on, you expect the total proportion outside those boundaries to be roughly about 30% (actually about $1-0.95^6 = 26.5\%$). If anything your coins are too fair!

If you had an a priori fixed stopping time, the chance that at least one coin is outside the bounds at that moment isn't 5% but more than 5 times that.

2) successive values in an individual track over time are not independent but positively correlated (they're scaled cumulative sums), so when you see one of the lines stray over its bound, you expect it's a lot more likely to stay outside on the next few tosses than 5%. (so for example, the variability in the expected number of tosses outside will be higher than you expect). So you might see excursions outside the bounds that last somewhat longer than you may anticipate.

3) when you notice the pale blue line goes outside the bounds, you're looking at the most extreme case of six. When you're assessing whether that's further from 0.5 than you expect, keep that in mind (that you're looking at the most extreme of six coins). So when assessing how unusual this is, you could simulate sets of fair coins (six coins at a time, over 200 tosses). What's the average time outside the 95% bounds for the one coin that's outside the bounds the most? I imagine it's probably a lot more than 5%.

Here's a simulation for the highest (proportion of heads) of six fair coins over 200 tosses (red jagged line), and 99 repeated simulations of the same process (gray), along with a blue 95% two tailed normal theory bound ($0.5+1.96\times 0.5/\sqrt{t}$)

The first one (red) was outside the bounds at the end, as were a good fraction of the others (I didn't worry about counting, we'll see why when we get to item (4) below). Note also that quite a few of them (clearly more than 2.5%, remembering that the same thing is going on at the low side - it might be more like 10% or so each side) seem to spend a good deal of time above the blue line.

So perhaps that light blue line in your plot doesn't look very unusual.

4) your assessment that there's maybe something going on seems to be based on post-hoc investigation of the outcome, not an a-priori decision rule. When you look at an event that has already occurred (one of the coins was outside the 95% bounds more than 5% of the time) and say 'Hey, wow, what are the chances of that?' but compute (even intuitively) the chances as if that particular event had been specified before the observation, all the above discussion of 'more correct' calculations are nonsense - they grossly underestimate the probability that you'll see something that will make you say 'Hey, wow, what are the chances of that?'*, because all the events that might trigger the subsequent calculation of probability haven't been specified.

* For example if one of the coins had been outside the 65% bounds 80% of the time, that might perhaps make you say "hey, what are the chances of that?" as well. As might a number of other possible outcomes.

So after considering 1), 2) and 3), you still need to precisely specify your event of interest up front, and then you can work out the chances of it turning up on fair coins.

-