I have the following situation: a data frame with two dimensions x
and y
, with three "areas": an a-parametric distribution centered around (1,1)
, one centered around (3,3)
, and the point (1,3)
. This point is an outlier only in the 2D
space, and not in either dimension.
In this simple case, it is easy to locate it, e.g. based on distance.
But what if I have, say, 100000 rows and 300 columns (features)? any attempt at a density approximation fails miserably, and a distance calculation is too expensive. Clustering algorithms like dbscan
think most of the points are noise, and cannot classify them. PCA doesn't really work, because I find that in many of the examples I encountered the dimension cannot be reduced significantly (the contribution to the variance doesn't decrease so fast).
Trying to divide the space into cells fails in this case, too - even if each dimension is homogeneous, and I classify very simple as below or above the median, 2^300 is, well, irrelevant.
I looked at previous Q&A here but didn't really find anything suitable for this kind of dimensions. Any thoughts?