2
$\begingroup$

I have the following plot of data:

enter image description here

and I am trying to separate the main part of the data with the outliers that are far away from the main data (for example the data found at around x=250, around x=300, around x=150 and around x=50 which are clear by eye but I need an automatic way to do it). I cannot fit the data with a particular equation because the data do not look the same in every data set I have, but the outliers that I want to find are always visible by eye like in this plot. I tried to use k-means but the clusters it finds is the left peak and the right peak which is not what I want. Do you have any suggestions on how to separate the main data from the outliers? Maybe an unsupervised learning method? I am not very familiar with machine learning and I am using MATLAB to do this.

$\endgroup$
9
  • 1
    $\begingroup$ (+1) There's a bit of an art to this sort of thing. Any information you can share about the typical characteristics of your data, how they vary, how many points might appear in outlying groups, and so on, can be useful in identifying or developing good approaches. $\endgroup$
    – whuber
    Commented May 29, 2015 at 19:58
  • $\begingroup$ Usually the outliers are between 1-5% of the overall data and sometimes the second peak on the right is more prominent and sometimes it does not exist at all. $\endgroup$
    – AL B
    Commented May 29, 2015 at 20:03
  • 1
    $\begingroup$ Again, an outlier is data which do not fit. Somehow you need to fit something to evaluate how a point is or not outlying. What is your alternative? $\endgroup$
    – Brethlosze
    Commented May 30, 2015 at 8:31
  • 1
    $\begingroup$ I would be cautious of concluding that data that do not fit are outliers. I think a safer definition (albeit one that makes identifying outliers more difficult) is that outliers are data that come from a different data generating process. $\endgroup$ Commented May 30, 2015 at 15:25
  • 1
    $\begingroup$ Here is a very similar question I answered: stats.stackexchange.com/questions/152401/data-and-curve-fitting/…. Upon further inspection, it seems to be the same question by the same user! $\endgroup$ Commented May 30, 2015 at 15:40

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.