# Analysing periodicity in sparse time series

I am interested in looking for periodicities in a several day long recording of electrical activity. The traces present a very steady baseline over which, from time to time, some short events (300-500 ms) appear (hence the sparse in the title, although I am not sure it is the right term to use).

Now, the nature of the data prevents me to analyse the trace as a whole (~20 days recorded at 10KHz, my computer will definitely not handle that), so I just wrote a series of routines to find the interesting events, and export their date/time (also their shape, but that is another story) to a text file.

So, I end up with a series of times to analyse.

Just by looking at the raw trace what I can see is:

1. A ~2 hours periodicity in the events
2. The events appear in groups. That is, I have a series of 20-30 events in the course of a couple of minutes, then nothing for ~2 hours, then other 20-30 events and so on. Note that the number of events in a group is variable, can be 30 in one and 5 in the next one.
3. The periodicity is fairly obvious, but it's not perfect: events happen roughly at 2 hours interval, and the exact time may vary from day to day.
4. There may be superimposed periodicities. In particular, I see events at a specific time of the day, which may or may not fit in the abovementioned 2 hours periodicity.

What can I do to statistically determine the periodicity?

What I have tried to do is to make a frequency histogram of the events, binning every 10 minutes, which empirically seemed like a sensible bin size, and then looking at the ACF or the FFT of the counts, but only in a few cases something pops out with the ACF (nothing with FFT).

So, how would you analyse this type of problem?

Bonus question: in certain cases I have missing data (1 or 2 days missing for technical reasons). How would I account for that?

PS: I am using R, but any non-R solution will do as well!

EDIT

Here is some sample data to play with: http://dl.dropbox.com/u/11676289/exampletimes.txt

Here is a plot of the times:

and their histogram, in 10 minutes breaks

• 1. Do you have your data anywhere so people can look at it? 2. If the data is sparse enough, maybe FT (not FFT) is a good tool ($O(n_{sparse}^2)$ vs. $O(n_{all} \ln n_{all})$ ). 3. Do you care for the shape of the event or do you treat them as points / deltas with their intensity? Mar 27, 2012 at 11:37
• For the moment I don't care for the shape, I would like to make a frequency analysis first, then I will try and concentrate on shapes. I'm not sure I follow you on the FT... could you expand a little bit on that? I'll upload a sample of the data.
– nico
Mar 27, 2012 at 12:36
• FT - calculating it straightforwardly from its definition. This naive approach may be actually beneficial for very sparse data. Mar 27, 2012 at 13:25
• @nico -- a quick literature search shows that work on this kind of estimation is quite recent, e.g., Shao and Lii, Modelling non-homogeneous Poisson processes with almost periodic intensity functions. I don't have access to the paper, but the authors have a presentation which summarizes their results. No available implementation that I can find... This isn't the way I'd go -- I prefer Bayesian methods. If you like, I'll <cont'd>
– Cyan
Apr 1, 2012 at 3:01
• ... do some EDA, and if the data looks model-able (is that a real word?) in the following fashion, I'll try to work up some R code for either log-Gaussian Cox process estimation or scaled logit-Gaussian Cox process estimation with the covariance function from Lii's previous paper on almost periodic processes that dealt with continuous state space.
– Cyan
Apr 1, 2012 at 3:21

One can analyze how Fano factor changes with the window size. For all time windows it is $>1$, contrary to what one could expect for a periodic process (for a periodic process one should get $<1$ for time window related to its periodicity); for Poisson process it should be just $1$.