My Silhouette score decreases as number of clusters increase. I'm using scikit's kmeans algorithm on the modified white wine dataset from UCI. Here's the final dataset I'm using - https://drive.google.com/open?id=1goh97QZB3V0rJSn4amLC-jBb3upsaIlH
Code
df = pd.read_csv('whiteWineTwoClasses.csv', header=0)
numberOfColumns = len(df.columns)
numberOfAttributes = numberOfColumns - 1
X = df.iloc[:,0:numberOfColumns-1]
Y = df.iloc[:, numberOfColumns-1]
scaler = StandardScaler()
scaler.fit(X)
xtrans = scaler.transform(X)
def getNumbersForKmeans(X, numberOfClusters):
kmeans = KMeans(n_clusters=numberOfClusters, random_state=0)
kmeans.fit(X)
labels = kmeans.labels_
inertiaScore = kmeans.inertia_
silScore = metrics.silhouette_score(X, labels, metric='euclidean')
return inertiaScore, silScore
print(getNumbersForKmeans(xtrans, 3))
print(getNumbersForKmeans(xtrans, 10))
print(getNumbersForKmeans(xtrans, 20))
print(getNumbersForKmeans(xtrans, 50))
print(getNumbersForKmeans(xtrans, df.shape[0]-1))
Output is (Look at second column)
(43830.24610203885, 0.13157830778113577)
(31412.978722003416, 0.11319449812661529)
(26173.031185455613, 0.10728651926177515)
(19896.193919556117, 0.10876065019480499)
(2.5006777281554326e-07, 0.00010210332874983413)
I was of the understanding that when the number of clusters are ~ number of data points, silhouette score should be ~1
I've looked at other answers but none of them seemed to actually help here.