1
$\begingroup$

I am attempting to use the bag-of-words approach to examine a large text data set. I am experimenting with using spherical K-means to cluster either documents or terms with respect to the other. I have gotten some promising results. However, I am unable to determine a good strategy for choosing a number of clusters.

When using K-means clustering, I can plot the within-group sum of squares (SSW) and look for the "elbow" of the curve to identify the point of diminishing returns.

Since spherical K-means does not minimize distance, SSW does not seem like the correct metric. Is there some useful measure to determine when additional clusters will provide diminishing returns in spherical K-means?

$\endgroup$
2
$\begingroup$

You can compute the SSW metric yourself. It's not magic.

Alternatively, you can just normalize your data too unit length, then try regular k-means instead of spherical k-means.

Don't expect too good results. Skmeans never worked well for me, probably because k-means is too sensitive to outliers, and text data is always full of outliers.

$\endgroup$
  • $\begingroup$ Thank you. I agree that it is straightforward to compute SSW. However, it is not clear to me that this is the correct metric for skmeans, since skmeans does not minimize distance. I have edited the question to clarify that I am looking for the correct metric, rather than how to implement it. $\endgroup$ – David Bruce Borenstein Dec 2 '15 at 15:43
  • $\begingroup$ If you search around this site, you can find some questions that discuss how the skmeans objective is related to the squared Euclidean objective, in particular after you preprocess your data to have unit length. It is not much different to squared Euclidean. $\endgroup$ – Anony-Mousse Dec 2 '15 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.