# Analysing periodicity in sparse time series

I am interested in looking for periodicities in a several day long recording of electrical activity. The traces present a very steady baseline over which, from time to time, some short events (300-500 ms) appear (hence the sparse in the title, although I am not sure it is the right term to use).

Now, the nature of the data prevents me to analyse the trace as a whole (~20 days recorded at 10KHz, my computer will definitely not handle that), so I just wrote a series of routines to find the interesting events, and export their date/time (also their shape, but that is another story) to a text file.

So, I end up with a series of times to analyse.

Just by looking at the raw trace what I can see is:

1. A ~2 hours periodicity in the events
2. The events appear in groups. That is, I have a series of 20-30 events in the course of a couple of minutes, then nothing for ~2 hours, then other 20-30 events and so on. Note that the number of events in a group is variable, can be 30 in one and 5 in the next one.
3. The periodicity is fairly obvious, but it's not perfect: events happen roughly at 2 hours interval, and the exact time may vary from day to day.
4. There may be superimposed periodicities. In particular, I see events at a specific time of the day, which may or may not fit in the abovementioned 2 hours periodicity.

What can I do to statistically determine the periodicity?

What I have tried to do is to make a frequency histogram of the events, binning every 10 minutes, which empirically seemed like a sensible bin size, and then looking at the ACF or the FFT of the counts, but only in a few cases something pops out with the ACF (nothing with FFT).

So, how would you analyse this type of problem?

Bonus question: in certain cases I have missing data (1 or 2 days missing for technical reasons). How would I account for that?

PS: I am using R, but any non-R solution will do as well!

EDIT

Here is some sample data to play with: http://dl.dropbox.com/u/11676289/exampletimes.txt

Here is a plot of the times:

and their histogram, in 10 minutes breaks

• 1. Do you have your data anywhere so people can look at it? 2. If the data is sparse enough, maybe FT (not FFT) is a good tool ($O(n_{sparse}^2)$ vs. $O(n_{all} \ln n_{all})$ ). 3. Do you care for the shape of the event or do you treat them as points / deltas with their intensity? – Piotr Migdal Mar 27 '12 at 11:37
• For the moment I don't care for the shape, I would like to make a frequency analysis first, then I will try and concentrate on shapes. I'm not sure I follow you on the FT... could you expand a little bit on that? I'll upload a sample of the data. – nico Mar 27 '12 at 12:36
• FT - calculating it straightforwardly from its definition. This naive approach may be actually beneficial for very sparse data. – Piotr Migdal Mar 27 '12 at 13:25
• @nico -- a quick literature search shows that work on this kind of estimation is quite recent, e.g., Shao and Lii, Modelling non-homogeneous Poisson processes with almost periodic intensity functions. I don't have access to the paper, but the authors have a presentation which summarizes their results. No available implementation that I can find... This isn't the way I'd go -- I prefer Bayesian methods. If you like, I'll <cont'd> – Cyan Apr 1 '12 at 3:01
• ... do some EDA, and if the data looks model-able (is that a real word?) in the following fashion, I'll try to work up some R code for either log-Gaussian Cox process estimation or scaled logit-Gaussian Cox process estimation with the covariance function from Lii's previous paper on almost periodic processes that dealt with continuous state space. – Cyan Apr 1 '12 at 3:21

## 1 Answer

Fourier Transform may not be the best tool for analysis of such sparse (and spiked) data. Autocorrelation (perhaps after binning and rescaling) is probably the right tool. Also, one may try using recurrence plot.

However, the process does not look periodic at all. It looks more as two Poission processed - one for releasing spikes, and the second - for the spikes themselves. Take a look at Cox process or materials on spike trains (which exhibits similar behavior the the one from your data).

One can analyze how Fano factor changes with the window size. For all time windows it is $>1$, contrary to what one could expect for a periodic process (for a periodic process one should get $<1$ for time window related to its periodicity); for Poisson process it should be just $1$.

Also, you may try to look at the following: