7
$\begingroup$

It is generally known that 'jumps' in frequency data are difficult to estimate. In the current literature, many different techniques for estimating such jumps have been tested and often with satisfactory results. A summarizing paper about some of these techniques would be, for example, Riley, 2008.

However, all these techniques are concerning frequency data that 'floats', in the sense that the data returns to a mean $\alpha>0$. I'm interested in detecting outbreaks for frequency data where $\alpha = 0$.

A visualisation of this type of frequency data (graph from Brookmeyer & Stroup, 2003) would be:

frequency data with mean 0

Now I found that this sort of data is often considered in 'disease outbreak detection' literature. But I am not able to find good transformations, algorithms or estimation procedures at all.

This might be due to the fact that I am unsure about the name of this sort of frequency data.

The graph I showed above of Brookmeyer & Stroup is a CUSUM plot of 'floating' frequency data, so it's not the data itself. They state that if the CUSUM plot exceeds $h\sigma$, an alert is declared. This makes sense as the CUSUM is a transformation of the deviations from the mean. But in the case of $\alpha=0$ type frequency data, this technique can't be applied.

So, I have two questions:

  • What are known transformations (such as CUSUM for 'floating' frequency data) for this type?
  • What are well known and widely used detection algorithms for this type?

Any insights are highly appreciated.

Edit (some intuition)

Simple algorithms that work well in real-time might be some type of transformation that takes into account the mean or stdev. of the data. Even though such a transformations are easy to construct, it might be very difficult to separate 'real outbreaks' from normal observations. This because the mean and stdev. both tend to zero when taken over a certain window. E.g. a threshold such as $k \sigma$ or $k \mu$ will not be robust.

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Jean-Paul, I appreciate accepting my answer and awarding the bounty. I'm glad that you found my efforts worthwhile and hope that you're enjoying reading the referenced materials on this interesting topic, especially the book. $\endgroup$ – Aleksandr Blekh Mar 24 '15 at 21:59
1
+50
$\begingroup$

Since it seems that you have a time series here, I would recommend you to check my relevant answer for some ideas (time series clustering, entropy measures, anomaly detection) as well as links to further information. In addition, I would also consider the Fourier transformation approach.

UPDATE (based on discussion with the OP in comments below):

In regard to seeking analytical solutions to the problem, it seems to me that there exist several approaches: 1) apply CUSUM (or other) transformation to original data and then perform further analysis (detection); 2) use CUSUM to detect the abrupt changes; 3) use alternative algorithms.

Speaking about CUSUM transformations, it seems to be a topic of some debate - some researchers advise against it, warning about potential loss of inferential validity, while others disagree and still consider that approach useful. You may also find this paper relevant and useful.

Speaking about the selection of appropriate algorithms as well as theory behind the topic and corresponding methods, I highly recommend excellent comprehensive and freely available online book "Detection of Abrupt Changes: Theory and Application" by Basseville and Nikiforov.

In regard to the selection of software for implementing the abrupt change (anomaly) detection, I would recommend to explore several R packages that seem to offer required analytic functionality (in addition to visual) and support CUSUM and similar algorithms. In particular, take a look at the following packages: strucchange (see vignette / JSS paper), changepoint (see vignette), surveillance. The last one asks for special mentioning (considering insights of yours and of RegressForward's) as this package presents a framework and implements statistical methods for analysis of epidemic-like process phenomena (in various fields far beyond epidemiology). The project's home page and development are hosted at R-Forge, but the R package is available on CRAN. A vignette and related presentation slides are available. I hope that this update significantly improves my answer and is helpful.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Actually, most of those recommendations are not for the type of series that I'm interested in. Even the slides that you posted in another answer say that this class of frequency data (the one my question is about) should be detected using visual inspection. $\endgroup$ – Jean-Paul Mar 23 '15 at 18:19
  • $\begingroup$ @Jean-Paul: I was under the impression that you're talking about analytical approaches ("good transformations, algorithms or estimation procedures"), not visual exploratory ones. At least, that's how your question has been formulated: "What are known transformations ... for this type? What are well known and widely used detection algorithms for this type?" (emphasis mine). I'm sorry I couldn't be of more help. $\endgroup$ – Aleksandr Blekh Mar 23 '15 at 18:30
  • 1
    $\begingroup$ Yes, that's exactly what I'm looking for. So what I'm trying to say with my last comment is that the literature you posted does not cover the type of frequency data I am considering. Only the slides that you posted in another anwer did mention this type of data, but their solution was visual inspection. Therefore, they did not propose anything useful for detecting such jumps analytically. So indeed I am looking for analytical approaches, and the sources you mention didn't cover this type of data. $\endgroup$ – Jean-Paul Mar 23 '15 at 18:46
  • $\begingroup$ @Jean-Paul: I understand. So, are you saying that general approaches to anomaly detection cannot be applied to this type of data (with possible post-processing to account for the specifics)? $\endgroup$ – Aleksandr Blekh Mar 23 '15 at 18:51
  • 1
    $\begingroup$ The book is perfect! Didn't read a lot yet but it seems that many methods will be useful. Thank you very much. $\endgroup$ – Jean-Paul Mar 23 '15 at 22:15
2
$\begingroup$

What an interesting data set. I imagine your specific type of data do not represent 500 independent events simply... co-occurring, then returning to 0 of these events occurring for years on end. Similar to disease outbreaks, there is (probably) a triggering event that changes the state of the world to one in which success is possible, then 500 successes occur, then we return to nothing happening again.

Would you consider hurdle models to estimate the probability of this "outbreak" occurring and the size of the "outbreak"? Hurdle models indicate that the natural state of the world is 0 events. But if some hurdle is passed, then the state of the world is one where the number of events is drawn from a different distribution (which does not include zero). They are very similar to zero-inflated models (where the second distribution still includes zero). However, they do not model jump discontinuities.

I think this matches your description of $h\sigma$, and given nothing else, it might be worth investigating further.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Yes exactly: there is a triggering event that causes an outbreak (events are dependent). I just want to detect the outbreak as soon as possible. Thank you for the recommendation! I will look into 'hurdle models'. From your description it sounds like that direction of models might be quite insightful for my purpose. $\endgroup$ – Jean-Paul Mar 21 '15 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.