11
$\begingroup$

Does anyone know a good method to determine if clustering using kmeans is even appropriate? That is, what if your sample is actually homogenous? I know something like a mixture model (via mclust in R) will provide fit statistics for the 1:k cluster case, but it seems like all of the techniques to evaluate kmeans requires at least 2 clusters.

Does anyone know of a technique to compare the 1 and 2 cluster cases for kmeans?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
12
$\begingroup$

The gap statistic is a great way of doing this; Tibshirani, Hastie & Walther (2001).

http://stat.ethz.ch/R-manual/R-devel/library/cluster/html/clusGap.html - The relevant R package.

The idea is that it performs a sequential hypothesis test of clustering your data for K=1,2,3,... vs a null hypothesis of random noise, which is equivalent to one cluster. Its particular strength is that it gives you a reliable indication of whether K=1, i.e. whether there are no clusters.

Here's an example, I was inspecting some astronomy data a few days ago as it happens - namely from a transiting exoplanet survey. I wanted to know what evidence there are for (convex) clusters. My data is 'transit'

library(cluster)
cgap <- clusGap(transit, FUN=kmeans, K.max=kmax, B=100)
for(k in 1:(kmax-1)) {
    if(cgap$Tab[k,3]>cgap$Tab[(k+1),3]-cgap$Tab[(k+1),4]) {print(k)}; 
    break;
}

With the gap statistic you're looking for the first value of K where the test 'fails' i.e. the gap statistic significantly dips. The loop above will print such a k, however simply plotting cgap gives you the following figure:
enter image description here See how there's a significant dip in the Gap from k=1 to k=2, that signifies there are in fact no clusters (i.e. 1 cluster).

Improve this answer
$\endgroup$
1
  • $\begingroup$ how to do the same for hierarchical clustering with single linkage? Can you please explain FUN argument of clusGap? I ran the below line for hierarchical kmax=20 cgap <- clusGap(cluster_feat_base[,2:ncol(cluster_feat_base)], FUN=hclust, K.max=kmax, B=100) . But its giving an error saying Error in FUNcluster(X, kk, ...) : invalid clustering method 2 $\endgroup$
    – GeorgeOfTheRF
    Commented Dec 13, 2016 at 10:28
5
$\begingroup$

You may try also a more recent method: A. Kalogeratos and A.Likas, Dip-means: an incremental clustering method for estimating the number of clusters, NIPS 2012.

The idea is to use statistical hypothesis testing for unimodality on vectors containing the similarity/distance between one point and the rest of the points of the set. The testing is done using Hartigan-Hartigan dip test, Ann. Statist. 13(1):70-84.

The method starts with all the dataset as one cluster and incrementally splits it as long as the unimodality hypothesis is rejected (i.e. more than one clusters is present).

So this method would indicate whether there are more than one clusters in data (your question), but it may provide also the final clustering.

Here you can find some code in Matlab.

Improve this answer
$\endgroup$
0
$\begingroup$

Suppose I am considering the same example,

library(cluster)
cgap <- clusGap(transit, FUN=kmeans, K.max=kmax, B=100)
for(k in 1:(kmax-1)) {
    if(cgap$Tab[k,3]>cgap$Tab[(k+1),3]-cgap$Tab[(k+1),4]) {print(k)}; 
    break;
}

How can I subset elements of clusters corresponding to best clustering solution based on maximum gap statistics? So that I can use it for further analysis on each of the clusters.

I know there is a command called subset. There are no issues using this command when we have given the number of clusters we want. But how to subset it when we want to subset based on optimal k obtained using gap (in short, subsetting elements of clusters if there is a loop)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.