Poisson distribution and cluster analysis

Question

What type of statistical test I could apply in order to test if my dataset, which follow a Poisson distribution, contains clustered observations?

Here my dataset:

df = read.table(text =    Year Freq
 1975   10
 1976   12
 1977    9
 1978   14
 1979   14
 1980   11
 1981    8
 1982    7
 1983   10
 1984    8
 1985   12
 1986    9
 1987   10
 1988    9
 1989   10
 1990    9
 1991   11
 1992   12
 1993    9
 1994   10
 1995    8
 1996   12
 1997   11
 1998   13
 1999    7
 2000   13
 2001   10
 2002    9
 2003    8
 2004   13
 2005   15
 2006   11
 2007   10
 2008   11
 2009    9
 2010   10
 2011    8
 2012   11
 2013   10
 2014    6, header = TRUE)

I need to do a cluster analysis on these data in order to know if they are clustered.

UPDATE

Please see below what I tried to do:

# Determine number of clusters
mydata <- as.data.frame(df$Freq)
wss <- (nrow(mydata)-1)*sum(apply(mydata,2,var))
for (i in 2:10) wss[i] <- sum(kmeans(mydata,
                                     centers=i)$withinss)
plot(1:10, wss, type="b", xlab="Number of Clusters",
     ylab="Within groups sum of squares")

#perform k-means analysis
t = kmeans(mydata, 5)

Answer 1

(As noted in the comments, it is not quite clear what your intent is. However, as you requested more detail on the Poisson-process version, here it is.)

Your question asks whether your "data ... contains clustered observations?", and suggests that your data follows a Poisson distribution. The data appears to represent counts of number of events occurring per year, $n_t$ for $t=1,\ldots,T$, covering the $T=40$ year period from 1975 to 2014.

If we consider the (unobserved) events as a realization of a point process in time, then "clustered" would mean the events tend to occur in intermittent bursts. In this case, the standard null model is that events occur independently of one another, and follow a Poisson process. Under this null model, your data would follow a Poisson distribution.

A crude but simple measure of "clustering" is the index of dispersion $$D=\frac{\mathrm{var}(n)}{\mathrm{mean}(n)}$$ where $D=1$ for a Poisson distribution, and $D>1$ would be an indication of clustering.

Your data has $$\hat{D}=\frac{4.23}{10.23}=0.41$$ where the "hat" indicates an estimate based on the sample statistics.

This is significantly less than one, which would actually indicate that your events are relatively regular, also known as "anti-clustered".

This can also be seen from the fact that in 40 years of data, the minimum number of annual events was 6, whereas for a Poisson process some lower values would have been expected. For a Poisson distribution with parameter $\lambda=10.23\frac{\mathrm{events}}{\mathrm{year}}$ (the MLE), there would be a 91% chance of seeing a year with $n<6$ events in 40 years of data. (The probability would be even higher if events were clustered.)

Answer 2

If it is Poisson distributed, then it cannot be clustered at the same time.

Clustering assumes there is more than one process responsible for generating the data. A single Poisson distribution would be a single process, so no clusters.

I.e. the question is ill-posed.

Answer 3

As @Anony-Mousse said, you cannot use kmeans with just a single variable. Perhaps when you are thinking of "clustering", you would like to know if the frequency of events are clustered around particular years or time periods. If so, you're asking about whether there is central tendency or multi-modality.

Perhaps you can start here: Test for bimodal distribution