0
$\begingroup$

For a class project I am clustering stock time-series via K-Means. For this initial project, we are choosing a fixed number of clusters, 10, and will not be optimizing the number of clusters. I ran K-Means 100 times, to assess cluster quality, I ranked them by smallest Cost Function, $$COST = \sum_{i=1}^{10} \frac{1}{\lvert C_i \rvert}\sum_{x \in C_i} \left\lVert x - C_i \right\rVert ^2$$ where $C_i$ is the center of the $ith$ cluster and $\lvert C_i \rvert$ is the number of observations in $ith$ cluster.

However, I also ranked the clusters by largest Between Sum of Squares, SSB which is given by $SSB = SST - SSW$. Which, since $SST$ is constant for all clusters, this is essentially minimizing the Within Sum of Squares, SSW, $$SSW = \sum_{i=1}^{10} \sum_{x \in C_i} \left\lVert x - C_i \right\rVert ^2$$.

These two methods both minimize two very close functions, the sum of average squares within per cluster and the sum of the squares within.

However, I have gotten quite different rankings when ranking based on Minimum Cost vs Maximum SSB.

Is it okay for the difference of averaging to affect the rankings significantly, or is this likely computational error?

What are the qualitative differences in using each to assess cluster quality?

$\endgroup$
1
$\begingroup$

It's all a matter of choice of weighting.

The usual way is to weight each sample equally.

By normalizing with the size of the cluster, you let a small cluster (1 sample) have as much weight as a large cluster... But small clusters are easy to "optimize", so that cost function will likely prefer degenerate results with tiny clusters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.