0
$\begingroup$

I have around 15-20 points every second, and I would like to detect outliers based on

-their density along x-axis , that means if I am using k-mean clustering then I specify that in x-direction max of a variance is tolerated and in y max of b is tolerated. and a << b for this case.

Hence I get elliptical clusters.

Moreover, I think that clustering algorithms based on density clustering will suit my problem or if you recommend any other please suggest.

In the following figure, the most I am interested in the points which are near to -10 at x-axis and I would like to retain them rest the ones >-12. I want to discard them, but I would need a general clustering method to do it for all the data, as the point (-10) in this specific case, changes from season to season.

Thank you for your time, and please let me know if any further info is required or you have any tips . It would be most welcome

Plot 2 Plot 1

Improve this question
$\endgroup$
  • 1
    $\begingroup$ Why not use an outlier detection algorithm? Clustering is not a good tools to detect outliers unless you are willing to put very stringent models on what the outliers can look like $\endgroup$ – user603 Oct 9 '14 at 12:55
  • $\begingroup$ Indeeed thats a nice idea, I didnt thought about that, would you recommend any specific outlier detection algorithm? $\endgroup$ – Omer Oct 9 '14 at 13:20
  • $\begingroup$ Sure, as long as you have two variables (X,Y), I would have a look at the OGK. It's computationally very light and easy to grasp (the link points to an R implementation with references therein). $\endgroup$ – user603 Oct 9 '14 at 13:28
  • 1
    $\begingroup$ k-means doesn't use thresholds a,b - so you are talking about some other algorithm, but not k-means. $\endgroup$ – Has QUIT--Anony-Mousse Oct 9 '14 at 19:04
  • $\begingroup$ Yes, you are right! Because k means gives both x and y dimension the same weightage, In my case, I would like to give my x-axis more weightage and y axis come. I cannot completely ignore one axis and when I give both of the axis same weight, during the transition, when the data is not concentrated in either of them, it gives poor results. $\endgroup$ – Omer Oct 10 '14 at 10:02
1
$\begingroup$

Why do you insist on using clustering?

To me it sounds as if simple k-nearest-neighbor distance (and in fact, nearest-neighbor distance) should just work for you. And that is a simple as it can get.

Note that it is trivial to add weights $\Omega=\{\omega_1,\ldots,\omega_d\}$ into Euclidean distance:

$$ \text{Euclidean}_\Omega(x,y) := \sqrt{\sum_{i=1}^{d} \omega_i(x_i-y_i)^2} $$

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your suggestion, the problem is when I apply KNN during the transition phase( i.e. when the concentration of data shifts from lets say in this case -10 to another number like -30, I would like to detect this rate of transition for which I need to give weightage to the x and y axis respectively. I am looking for something similar as you suggested but in which I am able to give the weightage to x and y axis for the decision. $\endgroup$ – Omer Oct 10 '14 at 10:00
  • $\begingroup$ You can use weightage for distance computations easily. Use weighted distance functions. $\endgroup$ – Has QUIT--Anony-Mousse Oct 10 '14 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.