0
$\begingroup$

just trying to understand the process of removing outliers in my data using the python Scikit's DBSCAN function.

As an example, given aDataFrame of data, which includes both my target and features, I do the following:

#Remove outliers
DB = DBSCAN(eps=0.5, min_samples = 10)
DB.fit(aDataFrame)
print (Counter(DB.labels_), '\n')  
filteredData = aDataFrame[DB.labels_ !=-1]

I'm seeing about 50% of my data removed as outliers, where I'd guess typically you'd see about 5% as outliers. (So I must be doing something wrong).

Should targets(labels) be included in the data passed to DBSCAN? Are there rule of thumbs you use when choosing good values for DBSCAN parameters?

$\endgroup$

closed as unclear what you're asking by Nick Cox, kjetil b halvorsen, Peter Flom Mar 24 '18 at 13:58

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ For outlier detection, you should rather use an outlier detection approach like Local Outlier Factor, or LoOP, or kNN outlier. LOF is by the DBSCAN authors, based on similar ideas but better at this task than clustering 'leftover' noise points. $\endgroup$ – Anony-Mousse Mar 24 '18 at 19:51
  • $\begingroup$ Apart from that, just increase epsilon to get less noise. $\endgroup$ – Anony-Mousse Mar 24 '18 at 19:51
1
$\begingroup$

There are no universal rules of thumb for DBSCAN parameters. What values "work" will depend on your data, in particular the distances between points. The curse of dimensionality can be a major problem for DBSCAN. (Well, in fact it can be a major problem for any clustering algorithm.)

Take a look at the distribution of pairwise distances. If 90% of your pairwise distances are 1 or larger, then setting eps=0.5 will guarantee that few if any clusters are found. Similarly, consider the size of your dataset and likely numbers of clusters when setting min_samples.

If you have targets in your data, then I wonder whether you should truly be doing clustering, which assumes that there are no targets - clustering is an example of unsupervised learning. Of course, you could include the targets as just another feature, and potentially scale the distance component between different target values to give the target higher weight. But if you just want to cluster similar targets together, it doesn't really make a lot of sense to use all the other attributes.

The Wikipedia description of DBSCAN is quite good and may help you in setting parameters.

$\endgroup$
  • $\begingroup$ Thank you very much Stephan, I'll give it more thought in that direction, marking as answered. $\endgroup$ – Rob Mar 24 '18 at 11:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.