3
$\begingroup$

The Validity Index "Pseudo F" is described as:

(between-cluster-sum-of-squares / (c-1)) / (within-cluster-sum-of-squares / (n-c))

with c beeing the number of clusters and n beeing the number of ovservations. It measures the seperation between all the clusters and should be high.

I want to apply K-Means for a large dataset. (about 3000 observations with about 200 values each).

When computing the Pseudo F for say cluster sizes 2 till 20, it decreases continously with the increase in the number of clusters.

How should I interpret these values? Possibilities might be.

  • The validity index is inappropriate for this type of clustering
  • K-Means is not very applicable for this dataset
  • The clustering solution with 2 clusters it the best

WSS of cluster:

enter image description here

$\endgroup$
  • $\begingroup$ This index is widely known as Calinski–Harabasz criterion. I recommend you read posts on this site which mention this index. $\endgroup$ – ttnphns Dec 10 '13 at 8:33
1
$\begingroup$

Psuedo F describes the ratio of between cluster variance to within-cluster variance. If Psuedo F is decreasing, that means either the within-cluster variance is increasing or staying static (denominator) or the between cluster variance is decreasing (numerator).

Within cluster variance really just measures how tight your clusters fit together. The higher the number the more dispersed the cluster, the lower the number the more focused the cluster. Between cluster variance measures how seperated clusters are from each other.

K-means objective is to minimize within cluster variance (necessarily maximizing between cluster variance). So the way that you can interpret this is: as the number of clusters increase the within cluster variance increases, making the actual clusters more dispersed/less compact and therefore less effective (and potentially closer to other clusters).

With that said, all of your interpretations are possible. But before going ahead and writing off k-means, you should try looking at the elbow method (plot # of clusters vs. between-group variance divided by total group variance) - if there's no elbow in the plot it's usually a good sign that k-means won't provide useful results (or at least that's my litmus test).

Here's an example of the elbow method using R code (from http://www.statmethods.net/advstats/cluster.html). Where "mydata" is your data.

# Determine number of clusters
wss <- (nrow(mydata)-1)*sum(apply(mydata,2,var))
for (i in 2:20) wss[i] <- sum(kmeans(mydata,centers=i)$withinss)
plot(1:20, wss, type="b", xlab="Number of Clusters",
ylab="Within groups sum of squares")
$\endgroup$
  • $\begingroup$ I tried out the elbow method and added results to the first post. When I extend the kmeans function with multiple random runs, the graph gets even more smooth. WSS decreases slightly with the last clusters and so between cluster variance increases only slightly. The denominator stays almost the same since the increase of c has almost no influence since n is much bigger. The numerator decreases much with the increase of clusters sincen a change of c has a big impact here. The ratio of wss to total variance is only about 20% can I conclude from this that the model might not be appropriate,too? $\endgroup$ – user2974776 Dec 9 '13 at 19:59
  • $\begingroup$ Does 2 clusters make sense? Reviewing the cluster means, can you interpret what makes these two groups unique? If the answer is yes, there's nothing wrong with that. However, if you feel there's something missing... time to try out another clustering method. R provides a vast range packages for clustering, some of which may even be specific to your domain expertise. The task view on clustering provides a good listing of packages available. cran.r-project.org/web/views/Cluster.html $\endgroup$ – Brandon Bertelsen Dec 9 '13 at 20:11
  • $\begingroup$ 2 does not make much sense, from the context it should be like 8 to 12 clusters. So probably I'll have a look at some other methods, thanks for your help and advice! $\endgroup$ – user2974776 Dec 9 '13 at 20:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.