Frequency jump detection

Question

It is generally known that 'jumps' in frequency data are difficult to estimate. In the current literature, many different techniques for estimating such jumps have been tested and often with satisfactory results. A summarizing paper about some of these techniques would be, for example, Riley, 2008.

However, all these techniques are concerning frequency data that 'floats', in the sense that the data returns to a mean $\alpha>0$. I'm interested in detecting outbreaks for frequency data where $\alpha = 0$.

A visualisation of this type of frequency data (graph from Brookmeyer & Stroup, 2003) would be:

Now I found that this sort of data is often considered in 'disease outbreak detection' literature. But I am not able to find good transformations, algorithms or estimation procedures at all.

This might be due to the fact that I am unsure about the name of this sort of frequency data.

The graph I showed above of Brookmeyer & Stroup is a CUSUM plot of 'floating' frequency data, so it's not the data itself. They state that if the CUSUM plot exceeds $h\sigma$, an alert is declared. This makes sense as the CUSUM is a transformation of the deviations from the mean. But in the case of $\alpha=0$ type frequency data, this technique can't be applied.

So, I have two questions:

• What are known transformations (such as CUSUM for 'floating' frequency data) for this type?
• What are well known and widely used detection algorithms for this type?

Any insights are highly appreciated.

Edit (some intuition)

Simple algorithms that work well in real-time might be some type of transformation that takes into account the mean or stdev. of the data. Even though such a transformations are easy to construct, it might be very difficult to separate 'real outbreaks' from normal observations. This because the mean and stdev. both tend to zero when taken over a certain window. E.g. a threshold such as $k \sigma$ or $k \mu$ will not be robust.

Answer 1

Since it seems that you have a time series here, I would recommend you to check my relevant answer for some ideas (time series clustering, entropy measures, anomaly detection) as well as links to further information. In addition, I would also consider the Fourier transformation approach.

UPDATE (based on discussion with the OP in comments below):

In regard to seeking analytical solutions to the problem, it seems to me that there exist several approaches: 1) apply CUSUM (or other) transformation to original data and then perform further analysis (detection); 2) use CUSUM to detect the abrupt changes; 3) use alternative algorithms.

Speaking about CUSUM transformations, it seems to be a topic of some debate - some researchers advise against it, warning about potential loss of inferential validity, while others disagree and still consider that approach useful. You may also find this paper relevant and useful.

Speaking about the selection of appropriate algorithms as well as theory behind the topic and corresponding methods, I highly recommend excellent comprehensive and freely available online book "Detection of Abrupt Changes: Theory and Application" by Basseville and Nikiforov.

In regard to the selection of software for implementing the abrupt change (anomaly) detection, I would recommend to explore several R packages that seem to offer required analytic functionality (in addition to visual) and support CUSUM and similar algorithms. In particular, take a look at the following packages: strucchange (see vignette / JSS paper), changepoint (see vignette), surveillance. The last one asks for special mentioning (considering insights of yours and of RegressForward's) as this package presents a framework and implements statistical methods for analysis of epidemic-like process phenomena (in various fields far beyond epidemiology). The project's home page and development are hosted at R-Forge, but the R package is available on CRAN. A vignette and related presentation slides are available. I hope that this update significantly improves my answer and is helpful.

Answer 2

What an interesting data set. I imagine your specific type of data do not represent 500 independent events simply... co-occurring, then returning to 0 of these events occurring for years on end. Similar to disease outbreaks, there is (probably) a triggering event that changes the state of the world to one in which success is possible, then 500 successes occur, then we return to nothing happening again.

Would you consider hurdle models to estimate the probability of this "outbreak" occurring and the size of the "outbreak"? Hurdle models indicate that the natural state of the world is 0 events. But if some hurdle is passed, then the state of the world is one where the number of events is drawn from a different distribution (which does not include zero). They are very similar to zero-inflated models (where the second distribution still includes zero). However, they do not model jump discontinuities.

I think this matches your description of $h\sigma$, and given nothing else, it might be worth investigating further.