How can I measure the effects certain events have on the frequency of other events over time?

Question

EDIT: added more details following @kjetil comments

I have the following problem:

• I monitor one stream of events of type A - those events can be considered instantaneous.
• I also monitor additional streams of events of types B1, B2,... Bn - those have a time interval associated with them.
• I want to determine if the frequency of type A events increases significantly during any specific B types (in other words, I want to determine exactly which B types have a significant positive influence during the intervals of their events)
• Many instances of the same type of event can occur simultaneously
• The effect is likely to be linear in the number of concurrent events of the specific B type
• There are probably many B types that don't effect A-events
• B types of events are not necessarily the only thing that influence type A events
• All events probably have some complex distribution that changes depending on the day/hour/week/month/previous events - not a simple Poisson
• I constantly monitor all streams - I want to discover when a B-type starts to influence the frequency of A-event and I would also want to know if at some point in the future it stopped influence it (and this should continue indefinitely))

For example:

• A-type events are power outages
• B1-type events correspond to opening and closing of air conditioners.
• B2-type events correspond to opening and closing of televisions.

Do you have any idea how I should approach this problem? (general references are fine)

Answer 1

Some ideas: For an initial analysis, you should make some plots. I suppose the main interest is in the possible dependence of intensity of A-events upon if the event occurs when a B-event is active or not. You could start with making a simple plot: on the y-axis you plot the cumulated number of A-events $n_A(t)$, on the x-axis you represent the time $t$, but you mark on the axis with a heavier line when an B-event is active. Then you can overplot with some smoother, and look to see if the slope seems to be different when B is active than when not. You could go further, make a smooth also of the derivative of the smoothed curve (with splines this can be done automatically, I think for the curve + its first derivative, a quintic spline is good. One possibility is to use the R package gss, but there are certainly others). When this is done, you can plot a histogram of (samples of) the smoothed derivative sampled at points $0,\delta t, 2\delta t, 3 \delta t, \dots $ within the two groups defined by B-event on or off.

Then for a model, you could base a simple model on the poisson process. Then we would need to go into how to calculate (or first define) a likelihood function for a poisson process. To avoid that, we do something simpler: Divide the t-axis (x-axis in the above plot) into small intervals. If the poisson process model is correct, the number of counts in each interval will be independent. Adjust so that each interval is completely B-event or not-B-event. count number of A-ponts in each interval. Fit a poisson regression (in R with the function glm()), with the interval length as an offset.

Of course, after doing this, you will need to investigate if the Poisson process assumption holds: Check for overdispersion, make a plot of interarrival times, which should be exponentially distributed (drop the times which takes the time between two A-points with different value of B), plot this separately for B and not-B, check if this seems to be exponential (you can do a density estimate).

For more ideas see the simple little book: "The statistical analysis of stochastic processes in time" by J K LIndsey, which is downloadable (legally!). google for it.