0
$\begingroup$

I dispose of data showing when a public authority issues different kinds of permits, say, planning permissions and liquor licences. I am interested in finding out whether the timing of one type of permission is done purposefully, e. g. one planning permission roughly every 14 days; whereas liquor licences are handed out at random dates.

A difficulty is that one type of permission is issued much more frequently over the period covered than the other.

My initial idea is to compare the variance in durations between the issuing of the permits for the different types, normalized for the number of events in each category.

Is this a sensible approach? I fear that this is too simple.

$\endgroup$
0
$\begingroup$

A wide sense stationary random process is defined as periodic if its autocorrelation function is periodic. The autocorrelation of a periodic function is, itself, periodic with the same period. [wiki]

You could define a discrete signal on your roughly periodic data as follows:

$x[t]=\left\{\begin{matrix} 0, \text{for no issuance on day }t\\ c_t, \text{for } c_t \text{ issuances on day } t \end{matrix}\right.$

Here, the days $t \in \{1,\ldots, n\}$.

Remove the mean of this signal to get $y[t]=x[t]-\text{mean}(x)$

The autocorrelation estimate is then: $R[k]=\dfrac{1}{n\sigma^2}\sum_{t=1}^{n-k} y[t]y[t+k]$

If you still see periodicity, then there's a good chance that your signal is periodic.

You could look at the Fourier amplitude spectrum of your autocorrelation function to pick out the peak corresponding to the period (frequency) of interest.


As for the suspected random signal: Poisson processes are usually used to model random arrivals / issuances. The inter-arrival time between events is exponentially distributed according to a rate factor $\lambda$.

For this signal, you could first exclude periodicity by running the autocorrelation/DFT and then perform tests to check if your data follows a Poisson distribution.

Your method of calculating the variance of interarrival times would give you the rate parameter of the process.

You could perform a statistical test to see if your data follows the Poisson distribution, although this could get fairly complicated.

The Poisson process has inter-arrival time, $T$, distributed exponentially as: $$p(T=t)=\lambda \exp(-\lambda t)$$ So your likelihood function, assuming that your inter-arrival time data are i.i.d samples: $$f(\lambda; D)=p(D|\lambda)=\prod_{p=1}^{N_{data}} p\left(T_p=t_p\right)=\lambda^{N_{data}} \exp\left(-\lambda \sum_p t_p\right)$$ Your loss function is then: $$l(\lambda; D)=-\ln f(\lambda; D)=\lambda \sum_p t_p - N_{data} \ln \lambda$$ Setting the derivative to zero gives: $$\dfrac{\partial l}{\partial \lambda}|_{\lambda=\hat{\lambda}}=0 \Rightarrow \dfrac{1}{\hat{\lambda}}=\dfrac{1}{N_{data}}\sum_p t_p$$

That is, your could take the mean of your inter-arrival times to estimate the rate parameter.

One crude way to check the validity of the Poisson assumption would be to estimate the variance of the inter-arrival time data as well. This should equal $1/\hat{\lambda}^2$, in theory. If it is too far off, then that indicates that assumptions on the process or data are wrong.


Note that these methods aren't rigorous and there are plenty of checks and alternatives that you would need to perform if you wanted to be more thorough. Some of these include:

  1. One doesn't use the autocorrelation function to exclude periodicities usually. But looking at a graph of the autocorrelation function should be able to guide a practitioner in the right direction along the decision tree.
  2. There are other possible random arrival process models that might explain the data better. The Poisson is simple and classic but not necessarily the best.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.