5
$\begingroup$

I will break my question appropriately to make it easier to explain. This is actually a short question but looks longer due to my plots so I apologize for that. Any input would be greatly appreciated.

Setup

I have a few locations that I am monitoring with sensors. The dataset that I have is a set of events that the sensors measured at different locations. The measurements themselves look like the following:

enter image description here

First take

I wanted to figure out if any of the events recorded by these sensors were correlated so I ended up computing the cross-correlation of these series (plotted below). The plot shows me that there seems to be some correlation between the sequences V3 and V10 but nothing to show a strong correlation. So I ended up revising my approach: I used a window to observe the events.

enter image description here

I took a time window of 40 seconds and plotted the time series again along with their cross-correlation with each other. This time, it gave me something encouraging. I can see a perfect correlation between 3 and 10 and 3 and 6.

enter image description here

and its corresponding cross-correlation plots.

enter image description here

Question

It looks like I should not be computing the cross-correlating the entire timeseries but rather only across a small window. This makes sense because I am not really looking to see if the entire time series is the same anyways. What I am really looking for is if there is a correlated event. If we assume that events beyond a certain time period need not necessarily be correlated then we can perhaps compute the cross-correlation for a sliding window that would be more accurate and notify only if it is greater than 0.8.

Well, at this point, I feel like I am reinventing the wheel and am hoping there are standard techniques to do this. Most of my data points would be zero with a sudden burst now and then. In that case, how can I make I conclusions (that there is indeed a correlation) stronger? I head about FFTs and DFTs. Would they be of any use here? Any comments?

$\endgroup$
16
  • $\begingroup$ Thank you for the pointer. I thought this was more of a signal-processing related question and have definitely seen questions like this on SO before. Ex: stackoverflow.com/questions/1289415/… I did not come across many DSP people on Cross-Validated but I could be wrong. $\endgroup$
    – Legend
    Commented Aug 7, 2011 at 22:57
  • $\begingroup$ maybe you should explain what you are trying to do in terms that are not technical. do you expect to see the same event at different places at the same time, or the same pattern of events at different places at different times, or what? it's not at all clear, because you are jumping into a (possibly incorrect solution) without explaining what on earth is actually important. $\endgroup$
    – andrew cooke
    Commented Aug 7, 2011 at 22:57
  • $\begingroup$ also ffts and dfts are just implementation details. you really need to clarify what you want first. $\endgroup$
    – andrew cooke
    Commented Aug 7, 2011 at 22:59
  • $\begingroup$ @Legend : The question you link at is from before the crossvalidated-era... As the policy on SO got a whole lot stricter, I rather act before a bunch of others comes to downvote and close your question as off topic. It has happened before... $\endgroup$
    – Joris Meys
    Commented Aug 7, 2011 at 23:02
  • 1
    $\begingroup$ in seismology you have something similar. an earthquake is detected at different times around the world, and the time differences depend on where it was. so no two events have the same shift. identifying which events belong together involves iterating with a model of where the earthquake occurred, and seeing what events fit. do you have any physical "meaning" behind the delays in your case? $\endgroup$
    – andrew cooke
    Commented Aug 7, 2011 at 23:24

1 Answer 1

4
$\begingroup$

OK, thanks for clarifying this in comments. I'll try explain things more clearly here.

A cross correlation is really just a way of seeing what shift between two sets of data aligns best. So if you have the same "pattern" at different sites, but the times are different because the clocks are badly adjusted (say), then a cross correlation will show a strong peak at the shift that corresponds to the time difference.

So you can use cross-correlation to find the time difference between two sites. Once you have that, you can correct the times of the events at one site so that they agree with the other. And then you can simply match up events by time.

Now, the next question is: over how long should you make the correlations? In general, the longer the better, because you have more events and so will get a stronger signal. But there is a trade-off, because you are assuming that the shift is constant over the length of the correlation, and if that is not true then you will get poor results.

To see that in more detail, lets assume that the time differences are due to clock errors, and that the clocks wander around slowly, so that over an hour they are pretty much constant (they don't agree, but they disagree by a fixed amount), but over a whole day they can have shifted backwards and forwards several times. In that case, correlating over a day is too long - better to divide the data into hour-long chunks and cross-correlate each chunk separately.

Hope that makes sense!

PS I still don't know what you're detecting. It's possible that you have something more complicated. For example, events could come in "groups", where the time shift for events within a single group is the same, but different groups have different shifts. In that case, what you need to do is identify the size (duration) of a group at one location and then use just the section of data that spans a single group, correlating it against other sites.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.