1
$\begingroup$

I have some problems in finding the outliers using clustering.

The data.frame is ~20000 observations and each row has mixed types of variables(numeric, nominal and binary). What I want to do is to detect the outliers by clustering.

I have calculated the dissimilarity matrix using daisy() function in R: 

diss = daisy(data,metric="gower")

And I know I can use pam() and hclust() functions to do the clustering. But how do I find the outliers after that?

Here is my R code to find the outliers from pam():

kmedoid = pam(diss,k=10,diss=T)
centers = kmedoid$id.med
distMat = as.matrix(diss)   
distances = rep(-99,20000)   
for (k in 1:20000) {   
  distances[k] = min(distMat[centers,k])   
}   
outliers = order(distances, decreasing=T)[1:5]   
outliers = data[outliers]   
outliers

I don't know whether it is correct, because the result seems to be pretty different each time when I tried different value of k in pam().

So the main question is: Once I have the "kmedoid" and "hc" calculated below, how do I find the outliers? 

kmedoid = pam(data,k=10,diss=T)
hc = hclust(data)

I did search Google, but there wasn't much info about this. I am not fluent in programming, so just using existing package and function in R would help me a lot:)

And is there any better method to find the outliers?

Thanks!

Improve this question
$\endgroup$
6
  • $\begingroup$ By definition of hclust algorithm, the nodes joined in at nearest the root/top of the tree have the largest distance from the others. Since distance is dissimilarity here, that means they're the most dissimilar to the others. That means they're more likely to be outliers (though that is not certain). Also, the choice of the linkage method in hclust will influence the shape of the tree and so the points that might look like outliers. $\endgroup$
    – conjectures
    Commented Jun 5, 2014 at 9:14
  • $\begingroup$ So, do I just check hc$height and choose the largest values to be outliers? $\endgroup$
    – user3710162
    Commented Jun 5, 2014 at 9:25
  • $\begingroup$ I used python for hclust, so not sure. But that's the gist of it. Large height does not imply outlier as some node will always be last to be added even if they really are all drawn from the same process. However, if a node really was an outlier you could reasonably expect it to get merged in at large height as it would not be similar to the other points. $\endgroup$
    – conjectures
    Commented Jun 5, 2014 at 10:05
  • $\begingroup$ Thanks a lot! I see. My problem now is that how I can do this in R? I don't know how to interpret the output of the hclust() function.. $\endgroup$
    – user3710162
    Commented Jun 6, 2014 at 3:10
  • $\begingroup$ You may want to look at outlier detection algorithms such as LOF and LoOP. Using randomized clustering methods such as k-means and PAM will yield different results every time, because the clusterings are different. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jun 6, 2014 at 13:02

1 Answer 1

Reset to default
2
$\begingroup$

To fill out my comments above with an example, in case it helps get a feel. Here are two clusterings of overlapping data from the iris dataset. The first has 50 nodes from one species and 2 from another - 'outliers' with respect to the main group. The second has all 50 from the same species.

enter image description here

enter image description here

There'll always be some nodes merged in relatively high and unless you know the true labelling it's not going to be clear whether relatively high = absolutely high. I think the asymmetry in the first graph compared to the second is interesting though.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Is there any method to rank the outlyingness from result of hclust? Since the dataset is large, it is hard to see outliers from just looking at the Dendrogram. $\endgroup$
    – user3710162
    Commented Jun 6, 2014 at 3:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.