3
$\begingroup$

I have a rather large dataset (200,000 rows x 85 cols) and I am trying to assess the cluster tendency prior to doing any cluster analyses. I have taken a subset of my data so I could run in R ( 6000 rows x 85 cols) and I am getting a very low Hopkins statistic value 0.05. I keep reading contradictory information on Hopkins values. Some say that above 0.5 is a "clusterable" data set, while anything below 0.5 is not and is uniformly distributed. Others say that anything above or below 0.5 is "clusterable" data. I am mainly trying to asses whether I should even try cluster analyses based on the cluster tendency, i.e. Hopkins statistic I am getting with my dataset. Thank you in advance.

$\endgroup$
4
$\begingroup$

I have also been confused about this contradictory information regarding Hopkins statistics.

In http://www.sthda.com/english/articles/29-cluster-validation-essentials/95-assessing-clustering-tendency-essentials/ it is said that

We can conduct the Hopkins Statistic test iteratively, using 0.5 as the threshold to reject the alternative hypothesis. That is, if H < 0.5, then it is unlikely that D has statistically significant clusters.

Put in other words, If the value of Hopkins statistic is close to 1, then we can reject the null hypothesis and conclude that the dataset D is significantly a clusterable data.

And there is also an example on iris dataset using get_clust_tendency() function that shows that for highly clusterable dataset the Hopkins statistic is 0.818, but for random dataset 0.466. However, if you repeat their analysis, you would actally get 0.182 for clusterable dataset and 0.534 for random dataset.

This suggests that the get_clust_tendency() function used has been changed (when? why?) s.t. now the Hopkins statistic is computed as $1-H$, where (definition from Wikipedia)

$ H=\frac{\sum_{i=1}^m{u_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \, $,

$u_i$ - nearest neighbour distances from uniformly generated sample points to sample data from given dataset,

$w_i$ - nearest neighbour distances within sample data from given dataset.

Thus, for clusterable datasets Hopkins statistic is close to 0, if computed using get_clust_tendency() function from factoextra package.

$\endgroup$
  • $\begingroup$ Sounds like a bug in factoextra, doesn't it? $\endgroup$ – Has QUIT--Anony-Mousse Mar 18 '18 at 10:28
  • 1
    $\begingroup$ In the package manual it is said that Hopkins statistic close to 0 means that the dataset is clusterable. But the STHDA reference is misleading. Don't know if this is considered a bug. $\endgroup$ – Iden Mar 20 '18 at 5:37
  • $\begingroup$ Mathematically, that is equivalent, and it is fairly common to see both in use. Probably with a small naming difference: "clustering tendency" IMHO means that larger values should be "more clustered". But as seen in my answer, the test is for uniformity. So it can actually only tell you when data is suspiciously uniform. $\endgroup$ – Has QUIT--Anony-Mousse Mar 20 '18 at 6:56
  • $\begingroup$ Wikipedia has incorrect information in the entry for Hopkins statistic. You need to just ignore the Wikipedia and go to some credible sources. $\endgroup$ – Geoffrey Anderson Feb 27 '19 at 22:28
2
$\begingroup$

Hopkins is a pretty extreme test for uniform distributions.

It's naive to assume that data will cluster, just because it has a tendency - the test is mostly useful to detect uniform data.

The problem is that it doesn't imply a multimodal distribution. A single Gaussian will have a "clustering tendency" according to Hopkins test. But running cluster analysis on a single Gaussian is pointless. The best result is "everything is the same cluster". It just tested that a Gaussian is not uniform.

Nevertheless, I would expect a small value to indicate the data looks uniform at least in the current normalization. If Hopkins indicates a uniform distribution, then you clearly can stop, there probably is some bad column ruining the analysis. But with the arguments above, the more sane interpretation is to use the opposite statistic 1-H and interpret this as a "normalized deviation from uniform data".

$\endgroup$
  • $\begingroup$ Very usefull - are there any alternative metrics that you would use for assessing cluster tendency ("clusterability")? $\endgroup$ – ALEX.VAMVAS Feb 17 at 22:06
  • $\begingroup$ Not that I know of besides actually trying different clustering algorithms. I also doubt that it's realistic to break this down to a single score - parts of a data set may be clusterable, even when large parts are not. In particular, local statistics won't work: consider a data set with two well separated uniform clusters. That should have maximum clusterability. Splitting it into clusters should make the value go to 0, they are just uniform now. $\endgroup$ – Has QUIT--Anony-Mousse Feb 24 at 9:25
0
$\begingroup$

Based on the source code for get_clust_tendency() it appear's that Iden's answer is correct and for some versions of the factoextra package, the hopkin's stat may be calculated as (1-H).

Check the code for the get_clust_tendency() function in your version of factoextra, if you see that the hopkin's stat is calculated as...

hopkins_stat = sum(minq)/(sum(minp) + sum(minq))

then this is the (1-H) version.

Contrary to the explanation given on Assessing Clustering Tendency,

sum(minq)

is actually the sum of the nearest neighbor distances for the real points, not the artificial ones.

With respect to the formula in Iden's answer

$$ H=\frac{\sum_{i=1}^m{u_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \ $$

The get_clust_tendency() may actually be returning

$$ H=\frac{\sum_{i=1}^m{w_i^d}}{\sum_{i=1}^m{u_i^d}+\sum_{i=1}^m{w_i^d}} \ $$

In the most recent version of the factoextra package this seems to have been corrected, see code line 110 in the source code where

hopkins_stat = sum(minp)/(sum(minp) + sum(minq))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.