2
$\begingroup$

I am trying to compute the silhouette coefficient in a clustered dataset which is 550k, but as the process is computational intensive I run out of memory and I cannot compute the silhouette coefficient.

So, I wonder if instead of validating the entire dataset, I take a sample (random or stratified) and then compute the silhouette coefficient on that sample?

$\endgroup$
1
  • 1
    $\begingroup$ In addition to Anony's excellent answer I might remark that there exists a version of Silhouette criterion which can be computed without computing the distance matrix. That is the "deviation from centroid" aka "simplified" Silhouette. If your data are continuous and clusters can reasonably have centroids (means), that version of Silhouette is a nice choice. $\endgroup$
    – ttnphns
    Sep 23, 2019 at 6:29

1 Answer 1

2
$\begingroup$

Yes, for a simple aggregate such as the Silhouette coefficient it is reasonable to compute it on a sample only. In many cases, it may also be fine to do the entire clustering only on such a sample (in particular if you do not know what works) rather than wasting the time to compute it on the full data.

In general there is little use in clustering huge data sets - it's expensive, and if you don't get additional knowledge compared to a sample, that was a waste of time. People tend to treat clustering as a toy exercise and scale it to huge data sets where the results don't yield any benefit... The smart way of using it is as a hypothesis generator; and a reasonably sized sample will be enough for that. You can then hypothesize what classes of customers you have etc. and - after human analysis and verification - label your data accordingly. Clustering never was something that you could fully automate, and it never will be.

$\endgroup$
2
  • $\begingroup$ (+1) I assume you mean 'never' and not 'need' in the final line. $\endgroup$
    – mkt
    Sep 23, 2019 at 17:08
  • 1
    $\begingroup$ Thanks. Stupid Gboard autocorrect. $\endgroup$ Sep 24, 2019 at 5:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.