# Poisson distribution and cluster analysis

What type of statistical test I could apply in order to test if my dataset, which follow a Poisson distribution, contains clustered observations?

Here my dataset:

df = read.table(text =    Year Freq
1975   10
1976   12
1977    9
1978   14
1979   14
1980   11
1981    8
1982    7
1983   10
1984    8
1985   12
1986    9
1987   10
1988    9
1989   10
1990    9
1991   11
1992   12
1993    9
1994   10
1995    8
1996   12
1997   11
1998   13
1999    7
2000   13
2001   10
2002    9
2003    8
2004   13
2005   15
2006   11
2007   10
2008   11
2009    9
2010   10
2011    8
2012   11
2013   10


I need to do a cluster analysis on these data in order to know if they are clustered.

UPDATE

Please see below what I tried to do:

# Determine number of clusters
mydata <- as.data.frame(df$Freq) wss <- (nrow(mydata)-1)*sum(apply(mydata,2,var)) for (i in 2:10) wss[i] <- sum(kmeans(mydata, centers=i)$withinss)
plot(1:10, wss, type="b", xlab="Number of Clusters",
ylab="Within groups sum of squares")

#perform k-means analysis
t = kmeans(mydata, 5)

• What do you mean "contains clustered observations"? Also, can you provide more context to the question. – Jon Oct 13 '16 at 15:45
• apologies. I just updated my question. – 2801001 Oct 13 '16 at 15:57
• are you looking for patterns in time? time series analysis? – oW_ Oct 13 '16 at 16:14
• apologies again, this is quite a new subject to me. I would like to know if my data (i.e. df$Freq) are clustered within years. – 2801001 Oct 13 '16 at 16:18 • Like everybody else, I'm completely confused by this question. The OP needs to elaborate a bit more about the high-level objectives of the analysis. What are you trying to understand? Avoiding references to jargon and software. – DJohnson Oct 13 '16 at 16:26 ## 3 Answers (As noted in the comments, it is not quite clear what your intent is. However, as you requested more detail on the Poisson-process version, here it is.) Your question asks whether your "data ... contains clustered observations?", and suggests that your data follows a Poisson distribution. The data appears to represent counts of number of events occurring per year,$n_t$for$t=1,\ldots,T$, covering the$T=40$year period from 1975 to 2014. If we consider the (unobserved) events as a realization of a point process in time, then "clustered" would mean the events tend to occur in intermittent bursts. In this case, the standard null model is that events occur independently of one another, and follow a Poisson process. Under this null model, your data would follow a Poisson distribution. A crude but simple measure of "clustering" is the index of dispersion $$D=\frac{\mathrm{var}(n)}{\mathrm{mean}(n)}$$ where$D=1$for a Poisson distribution, and$D>1$would be an indication of clustering. Your data has $$\hat{D}=\frac{4.23}{10.23}=0.41$$ where the "hat" indicates an estimate based on the sample statistics. This is significantly less than one, which would actually indicate that your events are relatively regular, also known as "anti-clustered". This can also be seen from the fact that in 40 years of data, the minimum number of annual events was 6, whereas for a Poisson process some lower values would have been expected. For a Poisson distribution with parameter$\lambda=10.23\frac{\mathrm{events}}{\mathrm{year}}$(the MLE), there would be a 91% chance of seeing a year with$n<6\$ events in 40 years of data. (The probability would be even higher if events were clustered.)

• thanks a lot. so my data are underdispersed (i.e. no clusters). I'll wait a bit before accept your answer. thanks – 2801001 Oct 15 '16 at 10:42
• do you know any method to test the significance of our index of dispersion? – 2801001 Oct 16 '16 at 15:09

If it is Poisson distributed, then it cannot be clustered at the same time.

Clustering assumes there is more than one process responsible for generating the data. A single Poisson distribution would be a single process, so no clusters.

I.e. the question is ill-posed.

As @Anony-Mousse said, you cannot use kmeans with just a single variable. Perhaps when you are thinking of "clustering", you would like to know if the frequency of events are clustered around particular years or time periods. If so, you're asking about whether there is central tendency or multi-modality.

Perhaps you can start here: Test for bimodal distribution