Validating cluster tendency using Hopkins statistic

Question

I have a rather large dataset (200,000 rows x 85 cols) and I am trying to assess the cluster tendency prior to doing any cluster analyses. I have taken a subset of my data so I could run in R ( 6000 rows x 85 cols) and I am getting a very low Hopkins statistic value 0.05. I keep reading contradictory information on Hopkins values. Some say that above 0.5 is a "clusterable" data set, while anything below 0.5 is not and is uniformly distributed. Others say that anything above or below 0.5 is "clusterable" data. I am mainly trying to asses whether I should even try cluster analyses based on the cluster tendency, i.e. Hopkins statistic I am getting with my dataset. Thank you in advance.

Answer 1

I have also been confused about this contradictory information regarding Hopkins statistics.

In http://www.sthda.com/english/articles/29-cluster-validation-essentials/95-assessing-clustering-tendency-essentials/ it is said that

We can conduct the Hopkins Statistic test iteratively, using 0.5 as the threshold to reject the alternative hypothesis. That is, if H < 0.5, then it is unlikely that D has statistically significant clusters.

Put in other words, If the value of Hopkins statistic is close to 1, then we can reject the null hypothesis and conclude that the dataset D is significantly a clusterable data.

And there is also an example on iris dataset using get_clust_tendency() function that shows that for highly clusterable dataset the Hopkins statistic is 0.818, but for random dataset 0.466. However, if you repeat their analysis, you would actally get 0.182 for clusterable dataset and 0.534 for random dataset.

This suggests that the get_clust_tendency() function used has been changed (when? why?) s.t. now the Hopkins statistic is computed as $1-H$, where (definition from Wikipedia)

$ H=\frac{\sum_{i=1}^m{u_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \, $,

$u_i$ - nearest neighbour distances from uniformly generated sample points to sample data from given dataset,

$w_i$ - nearest neighbour distances within sample data from given dataset.

Thus, for clusterable datasets Hopkins statistic is close to 0, if computed using get_clust_tendency() function from factoextra package.

Answer 2

Hopkins is a pretty extreme test for uniform distributions.

It's naive to assume that data will cluster, just because it has a tendency - the test is mostly useful to detect uniform data.

The problem is that it doesn't imply a multimodal distribution. A single Gaussian will have a "clustering tendency" according to Hopkins test. But running cluster analysis on a single Gaussian is pointless. The best result is "everything is the same cluster". It just tested that a Gaussian is not uniform.

Nevertheless, I would expect a small value to indicate the data looks uniform at least in the current normalization. If Hopkins indicates a uniform distribution, then you clearly can stop, there probably is some bad column ruining the analysis. But with the arguments above, the more sane interpretation is to use the opposite statistic 1-H and interpret this as a "normalized deviation from uniform data".

Answer 3

Based on the source code for get_clust_tendency() it appear's that Iden's answer is correct and for some versions of the factoextra package, the hopkin's stat may be calculated as (1-H).

Check the code for the get_clust_tendency() function in your version of factoextra, if you see that the hopkin's stat is calculated as...

hopkins_stat = sum(minq)/(sum(minp) + sum(minq))

then this is the (1-H) version.

Contrary to the explanation given on Assessing Clustering Tendency,

sum(minq)

is actually the sum of the nearest neighbor distances for the real points, not the artificial ones.

With respect to the formula in Iden's answer

$$ H=\frac{\sum_{i=1}^m{u_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \ $$

The get_clust_tendency() may actually be returning

$$ H=\frac{\sum_{i=1}^m{w_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \ $$

In the most recent version of the factoextra package this seems to have been corrected, see code line 110 in the source code where

hopkins_stat = sum(minp)/(sum(minp) + sum(minq))

Answer 4

Actually, as already mentioned in the previous answers, it is a matter of how Hopkins statistic has been calculated. The literature is has clear information on what value is expected. For example, Lawson and Jurs (1990) and Banerjee & Dave (2004) explains that you may expect 3 different results: 1) H = 0.5 (the dataset reveals no clustering structure) " in the formula, W always refers to the real data, and it is in the denominator) 2) H close to 1.0, a significant evidence that the data might be cluster-able. 3) H is close to 0, in this case the test is indecisive (data are neither clustered nor random)

based on the above info , you may find that: get_cluster_tendency(df, n, ..) provides the write calculation, whereas hopkins(df, ..) provides a reverse result as it might have been calculated based on 1-H. Cheers!

Answer 5

I have been pretty confused too with contradicting information available online. However if you are using pyclusterend library, the documentation says if hopkins value is less than 0.3, there are possibilities of cluster. To quote:

Hopkins test A statistical test which allow to guess if the data follow an uniform distribution. If the test is positve (an hopkins score which tends to 0) it means that the data is not uniformly distributed. Hence clustering can be useful to classify the observations. However, if the score is too high (above 0.3 for exemple); the data is uniformly distributed and clustering can’t be really useful for the problem at hand.