4
$\begingroup$

I am currently working with continuous data measured from different sensors (thermometers and voltmeters). I have a matrix whose columns represent the sensors and the rows are normalized measurements (one each second).

I have been asked to check if these sensors behave in the same way. Thus, I have used clusterization for the data. I have used DBSCAN with Euclidean and Manhattan distances. I have also used hierarchical clustering with Ward and average linkages and K-means using the elbow method to find the most suitable amount of clusters.

Now, I want to use the silhouette index to check which one of my clusterizations is the best.

My question is as follows: when I calculate the Silhouette index, should I use always euclidean distance?

As an example, in the hierarchical case:

from scipy.cluster.hierarchy import dendrogram, linkage, fcluster
from sklearn.metrics import silhouette_score
...

Clustering_H=linkage(data_state_normalized, 'average')
clusters=fcluster(Clustering_H, t=4, criterion="distance") 
silhouette_avg = silhouette_score(data_state_normalized, clusters, metric='euclidean')

I think that this is logical because it wouldn't make any sense to use euclidean as metric in the DBSCAN euclidean case and Manhattan in the DBSCAN Mankattan case. This way, all clusters will be measured in the same way. Is this right?

Besides, in the example presented, I cannot use average as metric in silhouette_score.

$\endgroup$
2
  • $\begingroup$ Logically, one would use the same distance function that was used in (or is implied in the algorithm of) the clustering done. But this is not a must. I can imagine doing clustering 1 by method using distance A, doing clustering 2 by method using distance B, and then comparing their results by Silhouette index using distance C. It may look eccentric, but there is no error with the decision. $\endgroup$
    – ttnphns
    Oct 25, 2023 at 15:08
  • $\begingroup$ I recommend you to check these "reflections" over internal cluster validity indices stats.stackexchange.com/a/358937/3277 $\endgroup$
    – ttnphns
    Oct 25, 2023 at 15:10

2 Answers 2

6
$\begingroup$

You can use any dissimilarity measure (or distance metric) that compares pairs of points, as described in [1,2 3]. It makes sense to use the same measure as used to construct the clusters as then they share the same criteria for inter/intracluster similarity.

If you use different measures, then you need to be careful to ensure the different dissimilarity measure used for Silhouette score makes sense and won't favor one cluster approach over the other, as you might find the Euclidean distance favoring the Euclidean clustering over the Manhattan distance.

The scikit learn silhouette metric score performs the averaging over all samples. If you want per sample they have silhouette_samples [3].

This function returns the mean Silhouette Coefficient over all samples. To obtain the values for each sample, use silhouette_samples.

References

  1. Peter Rousseeuw. "Silhouettes: A graphical aid to the interpretation and validation of cluster analysis". Journal of Computational and Applied Mathematics. Volume 20. November 1987. https://www.sciencedirect.com/science/article/pii/0377042787901257?via%3Dihub (Open access)
  2. https://en.m.wikipedia.org/wiki/Silhouette_(clustering)
  3. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. I will use then the sihouette method for the DBSCAN and K-means clusters. I will tell my boss that I cannot compare the hierarchical clusters since they don't use euclidean nor Manhattand distances $\endgroup$ Oct 25, 2023 at 17:11
5
$\begingroup$

(1) The distance function used should express as "correctly" as possible what distance "means" in the application in question. Any calculation based on distances, be it the clustering methods themselves or the Silhouette index (ASW for Average Silhouette Width in the following), will rely on having dissimilarity between objects correctly represented by the distance measure. If the distance measure is inappropriate, so is the resulting clustering, and also the Silhouette value. In particular, the ASW is inappropriate to compare different choices of distance.

To see this, as a thought experiment split your dataset randomly into five groups. Consider a distance measure that assigns distance 1 between any two observations from different groups and distance zero between any two observations from the same group. Most if not all clustering methods will reproduce these five groups at least if you fix the number of clusters to be five. The ASW will rate this at 1, i.e., maximum quality. Yet it doesn't mean a thing, because the distance measure is meaningless (and so is every index that is based on it).

(2) This means in particular that you should not pick a distance measure in order to achieve optimal clustering quality. Generally, assuming that there is a "true" distance between your observations, it may happen that this doesn't lead to a strong or convincing clustering. This would then be appropriate for the data you have - maybe there is no convincing clustering as your data are rather homogeneous. This can be valid and valuable information about your data. You wouldn't be served better by making the data look stronger clustered by applying an inappropriate distance measure.

(3) It is admittedly hard to say in most applications which distance measure is "correct", and the assumption that there is one unique "true" distance is debatable (although it is less debatable that you can define obviously inappropriate distance measures, see above). So I can understand why you'd want to look at and compare several different measures. Yet the data doesn't normally hold information about which distance measure is correct (unless there is secondary information that suggests it does, which is rarely the case in unsupervised settings; actually some kind of "supervision" would be required). This means that when choosing a distance measure, you should think hard about the meaning of the data and what it means to be "similar" or "distant" in terms of the subject matter. Looking at different clusterings in the data can only help with this in the sense that regarding subject matter information some of them may look more meaningful than others (which is not measured by ASW and neither by any other index that evaluates a clustering quantitatively).

(4) When discussing Euclidean vs. Manhattan two considerations are whether (a) the same change in differences on different variables should always result in the same change of the distance measure (as done by Manhattan) or whether changes between bigger differences in a variable are more important than the same changes between smaller differences in another variable (as they are for Euclidean due to the use of squares) and (b) whether data are in some way subjected to the rules of Euclidean geometry; e.g., whether it is required, for some reason, that the distance is rotation invariant, in other words, whether rotation of the coordinate system is a meaningful operation given the meaning of the variables (Euclidean is rotation invariant but not Manhattan). You also need to realise that it often makes a big difference to the distance whether you compute Euclidean or Manhattan on standardised data or not (usually this makes more of a difference than the choice of Euclidean or Manhattan itself), and there are different ways to standardise. All these are essentially different distances and the data themselves, disregarding their meaning, will not tell you what to use. Another option by the way is the Mahalanobis distance, which will assess the distance between observations relative to the variance-covariance structure in your variables (when it comes to clustering methods, this will correspond better to Gaussian mixture model-based clustering with flexible covariance matrices).

(5) If you think that the ASW is a more reliable clustering quality criterion than the objective function of $k$-means or objective functions on which other clustering methods are based, you may consider clustering the data by optimising the ASW in the first place, see F. Batool & C. Hennig (2021) Clustering with the Average Silhouette Width. Computational Statistics & Data Analysis. However, despite being co-author of that paper, I do not think that this is generally the case. As the principles of other clustering methods, also the ASW is based implicitly on its own "cluster concept" that will be appropriate in some applications and less appropriate in others, just as $k$-means or DBSCAN (but in different situations). So the ASW doesn't necessarily have more authority to decide what the best clustering is than any other clustering method in isolated use.

(6) Potentially problematic characteristics of the ASW are (a) that it can assess a clustering as bad where some clusters are very heterogeneous. Note that density-based clustering such as DBSCAN will leave rather heterogeneous point clouds together in the same cluster if there is no "density gap" separating subsets from each other. The ASW will in many situations not like this (unless such a cluster is really strongly separated from the rest of the data). Whether such clusters are desirable or not depends on how much within-cluster heterogeneity is tolerable in the specific application and more generally on what kind of clusters are required for what use. (b) The ASW can indicate to lump clusters together in order to achieve a very large between-clusters distance. For example, if you have four clusters that come as two pairs of clusters with the clusters within the pairs being fairly close together but the two pairs being strongly separated, the ASW will prefer a two cluster solution over a four cluster solution that is arguably better in most situations.

(7) That said, I do like to look at the ASW to compare different clusterings as an orientation rather than as an ultimately authoritative super-index. It is a sensible heuristic tool that has some disadvantages (no clustering approach or cluster quality index is perfect). When it comes to comparing distances and you have decided that, say, Euclidean is maybe a bit better than Manhattan to express the subject-matter meaning of distance in your application, it may give interesting information if a clustering that was computed based on the Manhattan distance could have a better or as good ASW value (computed using Euclidean distance) than a clustering computed based on Euclidean distances in the first place. This is rare but I have seen it happening. Still, the decision which distance to choose should be made on subject matter grounds, not by data-driven optimisation. Also, comparing different clusterings, I would always use quantitative validity (such as ASW) and visualisation and subject matter information (can the clustering be interpreted in a way that makes sense?) and stability experiments (e.g. as done by the clusterboot function in R-package fpc based on this paper of mine (sorry for self-advertisement;-). Also the question what the clustering will be used for should be kept in mind.

$\endgroup$
1
  • 2
    $\begingroup$ +1. Very thoughtful and thorough consideration. Thank you. $\endgroup$
    – ttnphns
    Oct 25, 2023 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.