1
$\begingroup$

I have the following situation: a data frame with two dimensions x and y, with three "areas": an a-parametric distribution centered around (1,1), one centered around (3,3), and the point (1,3). This point is an outlier only in the 2D space, and not in either dimension. In this simple case, it is easy to locate it, e.g. based on distance.

But what if I have, say, 100000 rows and 300 columns (features)? any attempt at a density approximation fails miserably, and a distance calculation is too expensive. Clustering algorithms like dbscan think most of the points are noise, and cannot classify them. PCA doesn't really work, because I find that in many of the examples I encountered the dimension cannot be reduced significantly (the contribution to the variance doesn't decrease so fast).

Trying to divide the space into cells fails in this case, too - even if each dimension is homogeneous, and I classify very simple as below or above the median, 2^300 is, well, irrelevant.

I looked at previous Q&A here but didn't really find anything suitable for this kind of dimensions. Any thoughts?

$\endgroup$

1 Answer 1

0
$\begingroup$

try to implement angle based outlier detection technique-

http://www.dbs.ifi.lmu.de/Publikationen/Papers/KDD2008.pdf ( original paper)

Implementation in R with simple explanation-

https://machinelearningstories.blogspot.com/2018/08/anomaly-detection-in-high-dimensional.html

$\endgroup$
1
  • $\begingroup$ Thanks! Looks like a very interesting read at least - I like ideas which are based on simple principles. Or like the saying goes: if it is not simple, it will simply not happen :) $\endgroup$
    – Omry Atia
    Commented Oct 5, 2018 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.