Validating cluster tendency using Hopkins statistic

I have a rather large dataset (200,000 rows x 85 cols) and I am trying to assess the cluster tendency prior to doing any cluster analyses. I have taken a subset of my data so I could run in R ( 6000 rows x 85 cols) and I am getting a very low Hopkins statistic value 0.05. I keep reading contradictory information on Hopkins values. Some say that above 0.5 is a "clusterable" data set, while anything below 0.5 is not and is uniformly distributed. Others say that anything above or below 0.5 is "clusterable" data. I am mainly trying to asses whether I should even try cluster analyses based on the cluster tendency, i.e. Hopkins statistic I am getting with my dataset. Thank you in advance.

We can conduct the Hopkins Statistic test iteratively, using 0.5 as the threshold to reject the alternative hypothesis. That is, if H < 0.5, then it is unlikely that D has statistically significant clusters.

Put in other words, If the value of Hopkins statistic is close to 1, then we can reject the null hypothesis and conclude that the dataset D is significantly a clusterable data.

And there is also an example on iris dataset using get_clust_tendency() function that shows that for highly clusterable dataset the Hopkins statistic is 0.818, but for random dataset 0.466. However, if you repeat their analysis, you would actally get 0.182 for clusterable dataset and 0.534 for random dataset.

This suggests that the get_clust_tendency() function used has been changed (when? why?) s.t. now the Hopkins statistic is computed as $1-H$, where (definition from Wikipedia)

$H=\frac{\sum_{i=1}^m{u_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \,$,

$u_i$ - nearest neighbour distances from uniformly generated sample points to sample data from given dataset,

$w_i$ - nearest neighbour distances within sample data from given dataset.

Thus, for clusterable datasets Hopkins statistic is close to 0, if computed using get_clust_tendency() function from factoextra package.

• Sounds like a bug in factoextra, doesn't it? Commented Mar 18, 2018 at 10:28
• In the package manual it is said that Hopkins statistic close to 0 means that the dataset is clusterable. But the STHDA reference is misleading. Don't know if this is considered a bug.
– Iden
Commented Mar 20, 2018 at 5:37
• Mathematically, that is equivalent, and it is fairly common to see both in use. Probably with a small naming difference: "clustering tendency" IMHO means that larger values should be "more clustered". But as seen in my answer, the test is for uniformity. So it can actually only tell you when data is suspiciously uniform. Commented Mar 20, 2018 at 6:56
• Wikipedia has incorrect information in the entry for Hopkins statistic. You need to just ignore the Wikipedia and go to some credible sources. Commented Feb 27, 2019 at 22:28

Hopkins is a pretty extreme test for uniform distributions.

It's naive to assume that data will cluster, just because it has a tendency - the test is mostly useful to detect uniform data.

The problem is that it doesn't imply a multimodal distribution. A single Gaussian will have a "clustering tendency" according to Hopkins test. But running cluster analysis on a single Gaussian is pointless. The best result is "everything is the same cluster". It just tested that a Gaussian is not uniform.

Nevertheless, I would expect a small value to indicate the data looks uniform at least in the current normalization. If Hopkins indicates a uniform distribution, then you clearly can stop, there probably is some bad column ruining the analysis. But with the arguments above, the more sane interpretation is to use the opposite statistic 1-H and interpret this as a "normalized deviation from uniform data".

• Very usefull - are there any alternative metrics that you would use for assessing cluster tendency ("clusterability")? Commented Feb 17, 2020 at 22:06
• Not that I know of besides actually trying different clustering algorithms. I also doubt that it's realistic to break this down to a single score - parts of a data set may be clusterable, even when large parts are not. In particular, local statistics won't work: consider a data set with two well separated uniform clusters. That should have maximum clusterability. Splitting it into clusters should make the value go to 0, they are just uniform now. Commented Feb 24, 2020 at 9:25
• Please note that it is important to raise the distances to the Dth power (D = dimension of the data). As of this data (Dec 2022) the get_clust_tendencey() function is not doing that. See this article: journal.r-project.org/articles/RJ-2022-055 Commented Jan 13, 2023 at 21:16

Based on the source code for get_clust_tendency() it appear's that Iden's answer is correct and for some versions of the factoextra package, the hopkin's stat may be calculated as (1-H).

Check the code for the get_clust_tendency() function in your version of factoextra, if you see that the hopkin's stat is calculated as...

hopkins_stat = sum(minq)/(sum(minp) + sum(minq))


then this is the (1-H) version.

Contrary to the explanation given on Assessing Clustering Tendency,

sum(minq)


is actually the sum of the nearest neighbor distances for the real points, not the artificial ones.

With respect to the formula in Iden's answer

$$H=\frac{\sum_{i=1}^m{u_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \$$

The get_clust_tendency() may actually be returning

$$H=\frac{\sum_{i=1}^m{w_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \$$

In the most recent version of the factoextra package this seems to have been corrected, see code line 110 in the source code where

hopkins_stat = sum(minp)/(sum(minp) + sum(minq))


Actually, as already mentioned in the previous answers, it is a matter of how Hopkins statistic has been calculated. The literature is has clear information on what value is expected. For example, Lawson and Jurs (1990) and Banerjee & Dave (2004) explains that you may expect 3 different results: 1) H = 0.5 (the dataset reveals no clustering structure) " in the formula, W always refers to the real data, and it is in the denominator) 2) H close to 1.0, a significant evidence that the data might be cluster-able. 3) H is close to 0, in this case the test is indecisive (data are neither clustered nor random)

based on the above info , you may find that: get_cluster_tendency(df, n, ..) provides the write calculation, whereas hopkins(df, ..) provides a reverse result as it might have been calculated based on 1-H. Cheers!

I have been pretty confused too with contradicting information available online. However if you are using pyclusterend library, the documentation says if hopkins value is less than 0.3, there are possibilities of cluster. To quote:

Hopkins test A statistical test which allow to guess if the data follow an uniform distribution. If the test is positve (an hopkins score which tends to 0) it means that the data is not uniformly distributed. Hence clustering can be useful to classify the observations. However, if the score is too high (above 0.3 for exemple); the data is uniformly distributed and clustering can’t be really useful for the problem at hand.

• The original statistic (see references in Wikipedia) is towards 1 if tgere are clusters, around 0.5 if there are uniform random (i.e., false) clusters, and is towards 0 if the structure is closer to lattice (i.e. no even random clusters). Commented Sep 7, 2022 at 18:08
• Yes, I have read that. but for pyclusterend, it seems to be this way. I have tested it too.
– s510
Commented Sep 8, 2022 at 8:10