I have a set of time series with events marked in the middle. Following the event there is a temporary dip in the series values followed by a peak so that the area under the curve is 0.

       event     response

This response is clearly visible in a plot of the average across all series, but the individual series are too noisy and varied to identify the response visually.

What is the best metric for estimating the statistical significance of this response? That is, how can I be sure it is not just a random artifact?


event details

The event is temporary and short enough to be considered a pulse

data context

This is step-count data in response to the delivery of a brief intervention. Example: subject gets a text message suggesting going for a run.

real data image

Though this effect is not sinusoidal as described above and a bit more exaggerated than the case I'm describing above, here is a plot illustrating a bump response in the averaged signal: example plot

  • $\begingroup$ Are you aware that you can post figures? That, I guess would make your question clearer. $\endgroup$ Mar 25 '15 at 13:25
  • $\begingroup$ @ChristophHanck, I can add a real figure if it helps, I thought the ascii got the point across and was cleaner though. $\endgroup$
    – 7yl4r
    Mar 25 '15 at 18:42
  • $\begingroup$ Is the huge peak we are seeing at 0 minutes since event the response you are looking for or is it the event? Also, your problem sounds similar to detecting responses to stimuli in certain FMRI experiments, so you might want to take a look into the respective literature. $\endgroup$
    – Wrzlprmft
    Mar 25 '15 at 20:45
  • $\begingroup$ @Wrzlprmft, that peak is the response; the event is in a different variable. $\endgroup$
    – 7yl4r
    Mar 25 '15 at 23:31

In Time series analysis, there is a methodology called intervention analysis originally developed by Box and Tiao in 1975. The intervention analysis can be done using transfer function modeling within arima framework. Below is the common type of intervention shapes from the article cited above. I think the one you are after is figure (f) below. You might have to modify transfer function to fit the shape of your specific curve. One you modify the curve and fit the model using Arima/transfer function framework, then you could do significance testing on your hypothesized curve shape. Another excellent book for transfer function modeling is Forecasting with Dynamic Regression Models by Pankratz.

Based on your edit to the question, it appears that you have a pulse input and a rapid decline in your response and settles to a new level for a while. I would go with figure (e).

On a side note, Box is an amazing applied statistician and was able to think about a problem and visually present it in a highly readable paper (even non statistician like me could understand it) and could be applied to a problem even 50 years later.

enter image description here

  • $\begingroup$ thanks for the great response. I've addressed your questions in an edit to the question. The response shape was an example and I understand how to design the transfer function, so let's assume (f) is what I want. It looks as though the linked paper may be exactly what I'm looking for, but it will take me a while to understand. Any additional guidance is greatly appreciated, of course. $\endgroup$
    – 7yl4r
    Mar 25 '15 at 18:53
  • $\begingroup$ Great, based on your chart, I would say it is more figure (e) or (d). I have add another reference which is very accessible. $\endgroup$
    – forecaster
    Mar 25 '15 at 20:38

To evaluate the significance of the peak you have to formulate your a priori knowledge on the individual time series you are measuring as a null hypothesis. A likely null hypothesis would be that your time series have the amplitude and frequency spectrum that they have and are uncorrelated to each other (and thus should not exhibit any peaks when averaged).

In this case, you can make use of time-series surrogates to generate a dataset that complies with your null hypothesis per construction. In particular, every prominent peak in the average of these surrogate time series would be due to chance.

Now, employ some measure for the prominence of your peak, such as its height or steepness in relation to the mean and standard deviation of your average. Apply this measure to both, your original peak in the average time series and the most prominent peak in the average time series of your surrogate data set. Repeat this with many surrogate data sets. If the prominence measure for your actual peak exceeds that for each of the surrogate data sets, you can assign your peak a significance based on the number of surrogate datasets you regarded.

If your peak is as prominent as the one you showed in your example (and unless you have a priori knowledge of your time series that is highly beneficial for the generation of such peaks), this approach might be overkill though. Also, if you want to estimate a very low significance level, you need a lot of surrogate datasets and thus a lot of computing time.

  • $\begingroup$ The peak in my example is the most prominent, other data shows the peak being within 1 std deviation of the mean. The significance seems to be in the fact that the signal will deviate 1 std deviation from the mean consistently for several minutes. Using surrogate data is a great idea; I'll give it a shot, thanks. $\endgroup$
    – 7yl4r
    Mar 26 '15 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.