EDIT: added more details following @kjetil comments

I have the following problem:

  • I monitor one stream of events of type A - those events can be considered instantaneous.
  • I also monitor additional streams of events of types B1, B2,... Bn - those have a time interval associated with them.
  • I want to determine if the frequency of type A events increases significantly during any specific B types (in other words, I want to determine exactly which B types have a significant positive influence during the intervals of their events)
  • Many instances of the same type of event can occur simultaneously
  • The effect is likely to be linear in the number of concurrent events of the specific B type
  • There are probably many B types that don't effect A-events
  • B types of events are not necessarily the only thing that influence type A events
  • All events probably have some complex distribution that changes depending on the day/hour/week/month/previous events - not a simple Poisson
  • I constantly monitor all streams - I want to discover when a B-type starts to influence the frequency of A-event and I would also want to know if at some point in the future it stopped influence it (and this should continue indefinitely))

For example:

  • A-type events are power outages
  • B1-type events correspond to opening and closing of air conditioners.
  • B2-type events correspond to opening and closing of televisions.

Do you have any idea how I should approach this problem? (general references are fine)

  • $\begingroup$ Why don't you post some actual data and I will try and apply time series methods including Intervention Detection procedures. $\endgroup$
    – IrishStat
    Dec 23 '12 at 19:29
  • $\begingroup$ @IrishStat - I'm trying to outline a work-plan for creating a program... I don't have data yet and even if I had, the program still needs to be robust enough to handle data that may behave considerably different. So basically I'm just trying to understand what are the most generic sensible approaches to this problem. More specifically, you've mentioned time-series methods and intervention detection... :) I'm looking into those, but I still didn't quite manage to understand how I should apply them in this case (started reading books on the time-series analysis just a few days ago) $\endgroup$ Dec 23 '12 at 20:06
  • $\begingroup$ Well you could always generate data according to a known/specified structure where some covariates were important and others not subject to your specification. The data could then be augmented with an error process , not necessarily indpendently distributed which could potentially mask the underlying DGF. You could also augment/perturb your output series with level shifts / time trends or holiday effects (pre and post ). This kind of data , although artificial , could teach you about how to deal with i.e. analyze real data. $\endgroup$
    – IrishStat
    Dec 23 '12 at 23:30
  • $\begingroup$ @IrisihStat - I get that in order to give a high quality robust answer I need to get a statistician and do a lot more analysis... but since that's not in the budget (just got a few CS researchers with bad CS research methodologies). I'd appreciate it if you could point to relevant topics/suggest references. Note that the program doesn't have to be perfect (and can't, because of the difference in data sources). Anything better than doing a z-test on the frequency of A during some B (using the mean and variance of A frequency) is a good place to start. In any case, thanks for your help! $\endgroup$ Dec 24 '12 at 7:08
  • $\begingroup$ Sorry but I can't help you do it wrong ...or less wrong. I would have no idea about these references that you request. $\endgroup$
    – IrishStat
    Dec 24 '12 at 13:34

Some ideas: For an initial analysis, you should make some plots. I suppose the main interest is in the possible dependence of intensity of A-events upon if the event occurs when a B-event is active or not. You could start with making a simple plot: on the y-axis you plot the cumulated number of A-events $n_A(t)$, on the x-axis you represent the time $t$, but you mark on the axis with a heavier line when an B-event is active. Then you can overplot with some smoother, and look to see if the slope seems to be different when B is active than when not. You could go further, make a smooth also of the derivative of the smoothed curve (with splines this can be done automatically, I think for the curve + its first derivative, a quintic spline is good. One possibility is to use the R package gss, but there are certainly others). When this is done, you can plot a histogram of (samples of) the smoothed derivative sampled at points $0,\delta t, 2\delta t, 3 \delta t, \dots $ within the two groups defined by B-event on or off.

Then for a model, you could base a simple model on the poisson process. Then we would need to go into how to calculate (or first define) a likelihood function for a poisson process. To avoid that, we do something simpler: Divide the t-axis (x-axis in the above plot) into small intervals. If the poisson process model is correct, the number of counts in each interval will be independent. Adjust so that each interval is completely B-event or not-B-event. count number of A-ponts in each interval. Fit a poisson regression (in R with the function glm()), with the interval length as an offset.

Of course, after doing this, you will need to investigate if the Poisson process assumption holds: Check for overdispersion, make a plot of interarrival times, which should be exponentially distributed (drop the times which takes the time between two A-points with different value of B), plot this separately for B and not-B, check if this seems to be exponential (you can do a density estimate).

For more ideas see the simple little book: "The statistical analysis of stochastic processes in time" by J K LIndsey, which is downloadable (legally!). google for it.

  • $\begingroup$ Thanks a lot! a few questions: (1) why use the derivative function and not just the average frequency of A-events? (2) assuming that counting the number of A in B and non-B intervals gives me 2 Poisson distributions - how do I compare the two? (your answer is helpful- but, I actually have additional limitations: (1) events probably depend on each other (Poisson?) (2) got an infinite automatic process (a monitoring program) (3) few B-events can happen in the same time (4) I can't really measure non-B intervals- I actually monitor many types of B-events). $\endgroup$ Dec 22 '12 at 10:19
  • $\begingroup$ Also, thanks for the book reference, I'll definitely check it out. $\endgroup$ Dec 22 '12 at 10:25
  • $\begingroup$ I think you should give some more details. How many different B-events are there? What do you measn by"I cannot really measure non-B" ¿Cant you o9bserve a-points when some B-event not are active? Again, we need more details! As for the monitoring, as soon as we have a working model we can look into it! Maybe the CRAN package strucchange? $\endgroup$ Dec 23 '12 at 0:27
  • $\begingroup$ Hi after giving it some thiught - that last comment (4) is wrong - I have thousands of A types and B types so I don't currently count A-non-B frequency - but with right implementation I can save all the frequencies. As for the package -maybe it can help but I still need to understand what type of analysis I should do... $\endgroup$ Dec 23 '12 at 3:33
  • $\begingroup$ Yes, but the situation is obviously much more complex than your originally simple description seems to apply, so we do need more details ... $\endgroup$ Dec 23 '12 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.