I'm using the method described in this paper for determining the optimal epsilon value for DBSCAN clustering in which a plot of the nearest neighbors is used:
However, the plots in the paper and other tutorials look like this: https://imgur.com/a/q00hE1u
And my plot looks like this: https://imgur.com/a/MHNJuNL
In short, their plot has a long shallow slope then a spike, which is supposed to indicate the optimal epsilon.
Mine has an immediate spike which shallows off. As it happens, my plot shallows off at between .5 and .7, which seem to give good results as the epsilon value, but I just want to be able to explain the difference in the shapes.
Here is a snippet of my code
tfidf_matrix = tfidf.fit_transform(texts) ... nbrs = NearestNeighbors(n_neighbors=2, metric='cosine').fit(tfidf_matrix) distances, indices = nbrs.kneighbors(tfidf_matrix) distances = np.sort(distances, axis=0) distances = distances[:,1] plt.plot(distances) plt.show()
I wonder if the difference has to do with the fact that I'm clustering texts using tf-idf cosine similarity? In the tutorials and the paper they're clustering some large continuous values that they normalize to between 0 and 1.
Additionally, does anyone have any good suggestions regarding evaluating DBSCAN clusters? Right now I'm experimenting with silhouette score, but I'm getting low scores (around 0.1). This seems wrong, though, since I can read the texts and see that the clusters are actually very good.