# Why are most of my points classified as noise using DBSCAN?

I'm using several clustering algorithms from sklearn to cluster some data, and can't seem to figure out what's happening with DBSCAN. My data is a document-term matrix from TfidfVectorizer, with a few hundred preprocessed documents.

Code:

tfv = TfidfVectorizer(stop_words=STOP_WORDS, tokenizer=StemTokenizer())
data = tfv.fit_transform(dataset)

db = DBSCAN(eps=eps, min_samples=min_samples)
result = db.fit_predict(data)
svd = TruncatedSVD(n_components=2).fit_transform(data)
// Set the colour of noise pts to black
for i in range(0,len(result)):
if result[i] == -1:
result[i] = 7
colors = [LABELS[l] for l in result]
pl.scatter(svd[:,0], svd[:,1], c=colors, s=50, linewidths=0.5, alpha=0.7)


Here's what I get for eps=0.5, min_samples=5:

Basically I can't get any clusters at all unless I set min_samples to 3, which gives this:

I've tried various combinations of eps/min_samples values and get similar results. It seems to always clusters areas of low density first. Why is it clustering like this? Am I maybe using TruncatedSVD incorrectly?

• Welcome to Cross Validated! Please take a moment to view our tour. Mar 29 '17 at 18:52
• The scatter plots show no trend but it could be that the variance is not constant. Mar 29 '17 at 19:04
• @MichaelChernick: This comment seems misplaced. What do you mean by trend and why do we care about it in this clustering application? If anything the scatter of the first two PC scores show one obvious cluster. DBSCAN does not examine within cluster variance or anything like that... Mar 29 '17 at 22:50
• Note that your probably should use DBSCAN with cosine distance rather than Euclidean distance here. Apr 2 '17 at 10:46

• Great, I am glad I could help. If you believe this post answers your question you could consider accepting the answer. $k$-means (probably) was able to find a clustering because it does not depend directly in a $\epsilon$-like factor like DBSCAN for the radius of its cluster; that doesn't mean that the clustering is sensible though. I have not used the sklearn's Python DBSCAN implementation so I cannot comment on its quality. Notice though that some implementations might do default pre-processing steps and that might accidentally affect their performance when applied. Mar 31 '17 at 19:27