1
$\begingroup$

My Silhouette score decreases as number of clusters increase. I'm using scikit's kmeans algorithm on the modified white wine dataset from UCI. Here's the final dataset I'm using - https://drive.google.com/open?id=1goh97QZB3V0rJSn4amLC-jBb3upsaIlH

Code

df = pd.read_csv('whiteWineTwoClasses.csv', header=0)
numberOfColumns = len(df.columns)
numberOfAttributes = numberOfColumns - 1
X = df.iloc[:,0:numberOfColumns-1]
Y = df.iloc[:, numberOfColumns-1]
scaler = StandardScaler()
scaler.fit(X)
xtrans = scaler.transform(X)

def getNumbersForKmeans(X, numberOfClusters):
  kmeans = KMeans(n_clusters=numberOfClusters, random_state=0)
  kmeans.fit(X)
  labels = kmeans.labels_
  inertiaScore = kmeans.inertia_
  silScore = metrics.silhouette_score(X, labels, metric='euclidean')
  return inertiaScore, silScore


print(getNumbersForKmeans(xtrans, 3))
print(getNumbersForKmeans(xtrans, 10))
print(getNumbersForKmeans(xtrans, 20))
print(getNumbersForKmeans(xtrans, 50))
print(getNumbersForKmeans(xtrans, df.shape[0]-1))

Output is (Look at second column)

(43830.24610203885, 0.13157830778113577)
(31412.978722003416, 0.11319449812661529)
(26173.031185455613, 0.10728651926177515)
(19896.193919556117, 0.10876065019480499)
(2.5006777281554326e-07, 0.00010210332874983413)

I was of the understanding that when the number of clusters are ~ number of data points, silhouette score should be ~1

I've looked at other answers but none of them seemed to actually help here.

$\endgroup$
3
  • $\begingroup$ Please post not the code (or not only it) but also results: cluster membership variable for each k, silhouette value for each k. You should also tell whether you standardized your variables in clustering or not. $\endgroup$
    – ttnphns
    Commented Oct 27, 2018 at 14:08
  • $\begingroup$ 1. Hey, I've posted silhoutte scores (Second column) I cannot paste cluster membership because it's 4800 row data. 2. By standardizing, do you mean this - xtrans = scaler.transform(X)? I've done this in the code. $\endgroup$
    – Sashank
    Commented Oct 27, 2018 at 22:38
  • $\begingroup$ I have similar issue. In my case as the number of clusters increases samples having positive silhouette score decreases and negative increases. Where you able to find what the issue is? $\endgroup$
    – deepguy
    Commented Dec 27, 2019 at 4:29

1 Answer 1

2
$\begingroup$

Your assumption on Silhouette is wrong:

The silhouette of a one elemental cluster leads to a division by zero (there is no average distance to other members of the same cluster). The authors defined the value of one-elemental clusters to be 0.

So when k -> N then Silhouette tends to 0 by definition.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.