1
$\begingroup$

Pardon the novice question, but I really don't know what I'm supposed to be searching for on this one.

I am monitoring a system for raw events. I can identify the "type" or "kind" of an event, but there is no "value" of an event other than it occurred. This could be summarized as a "count per hour", if necessary, but that's not part of the raw data. These events can occur as infrequently as once every 3 hours, or up to 5 times an hour.

What I need to be able to do is detect when these events are slowing down, as it indicates a problem with the upstream system. Any solutions I'd be comfortable using would need to be on-line, ie not batch processed once an hour. Is there a well-established way of doing this? Or do most people just figure out the count/hr vs. expected count/hr and call it good enough?

EDIT: These events are supposed to occur at a very regular rate. The kind of the event that occurs, however, is random (exact distribution is unknown and not under my control). Sometimes, however, certain kinds of these events stop happening. The time between events of the same kind might be dependent or independent, but time between events of any kind should be fixed, when everything is working correctly. Most kinds of events won't show up in my data (as in, all events of a certain kind are "filtered out"), so it might be hard to see the regularity of the events.

Improve this question
$\endgroup$
3
  • $\begingroup$ Look into process control and control charts, stats.stackexchange.com/questions/15364/… and since you need a control chart for counts mnaybe stats.stackexchange.com/questions/2154/… . Search this site for control chart $\endgroup$
    – kjetil b halvorsen ♦
    Commented Aug 26, 2017 at 19:41
  • $\begingroup$ so e.g. it is randomly chosen every minute which event happens and it can also be an event, that you don't see? did I understand it correctly? $\endgroup$
    – Tom83B
    Commented Aug 26, 2017 at 20:30
  • $\begingroup$ @Tom83B Correct. Not seeing an event happen means either no event happened, or "was filtered out". But, for the kinds of events I'm interested in, I should see all the events of that kind that occur. As in, some kinds of events are consistently not seen, while some kinds of events are seen every time they occur. So, we can use historical data about which event kinds happened when to predict how often events of a given kind should occur, right? If necessary, I can rework the system to record all kinds of events, but that's a trickier setup. $\endgroup$
    – xenrelay
    Commented Aug 26, 2017 at 20:35

1 Answer 1

Reset to default
0
$\begingroup$

Are the times between events independent from each other? What about looking at the times between them?

You can do a statistical test, whether the last $n$ times between events were drawn from the distribution the rest of the events satisfy.

Improve this answer
$\endgroup$
2
  • $\begingroup$ See if the edit in my question helps. $\endgroup$
    – xenrelay
    Commented Aug 26, 2017 at 20:17
  • $\begingroup$ I think this might be better as a comment rather than an answer? $\endgroup$
    – mdewey
    Commented Aug 26, 2017 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.