1
$\begingroup$

I'm in the middle of a result analysis for some clustering methods, doing quality tests for different clustering outputs coming from a singular input dataset where data preprocessing and cleaning methods are swapped.

So far, the clustering outputs from dataset where any outlier detection technique has been applied show a poor performance. Hence, I was wondering whether it's worth at all applying an outlier detection technique for clustering. My particular results say it isn't, but I'd like to know your opinions from a wider perspective.

If needed, the clustering methods used are: K-means, SOM maps and hierarchical clustering. Thanks!!

$\endgroup$
  • 4
    $\begingroup$ Have you looked at clustering using DBSCAN? It explicitly identifies outliers during clustering. $\endgroup$ – Stephan Kolassa Aug 3 '15 at 6:34
  • 1
    $\begingroup$ If you say: "have poor performance", what do you mean specifically? $\endgroup$ – Karel Macek Aug 3 '15 at 14:48
  • $\begingroup$ With poor performance, I mean low external validation measures, compared to clustering outputs coming from a dataset without outlier detection $\endgroup$ – formacero10 Aug 4 '15 at 12:10
  • $\begingroup$ Stephan, I didn't try that method specifically, it wasn't part of my research objectives; thanks for your suggestion $\endgroup$ – formacero10 Aug 4 '15 at 12:13
1
$\begingroup$

It really depends on your data, the clustering algorithm you use, and your outlier detection method. Consider the K-means algorithm. If your dataset has ``outliers", then the outliers can affect the result of clustering by shifting the cluster centers. Be careful to not mix outlier with noisy data points. Noise is a random effect on data and can appear in all directions. Outliers are single, mostly isolated data points that are far from the rest of the data.

If you do not have outliers, outlier detection can hurt your data by removing small clusters or removing only a part of a scattered noise.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

This is as much discussion as answer, and I doubt there is a single "answer".

I'm halfway though a clustering class for R on DataCamp with Dmitriy Gorenshteyn. I'm realizing that I've wasted many hours of my life trying to do clustering by trial an error.

To manage outliers it would be great to to omit / group the outliers into their own cluster so that just the meaningful ones remain, but this isn't possible since there's no similarity measure to unite outliers.

So, I think it makes more sense to split by the height you want (rather than the k groups you expect), and just keep the clusters with high similarity measures / relatively high counts. With hierarchical clustering you'll end up with a lot of "clusters" that are really single observations, and can be considered outliers.

It looks like DBSCAN is the real way to go (great suggestion by @stephan-kolassa).

It makes so much more sense to me now to realize that often most observations may not belong in a cluster.

That the distance metric (max, min, average) must impact on how outliers are included or excluded from clusters, but I'm not sure which measure would be better for managing outliers. Maybe I'll update my answer later, or someone else will provide a better answer in the future.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.