2
$\begingroup$

I am trying to cluster a very large set of data points, of roughly (20000, 100) shape. I could not run density based DBSCAN or SpectralClustering due to the enormous amount of memory required. However when I do K-Means (with appropriate K selected with the 'elbow' method) or GMM, I see that points that are visually far apart and should not be labeled in a single cluster are grouped together.

My guess is such clusterings are not good for sparse data. Any suggestion would be great.

I'm attaching the clustered scatter plot (X, Y are normalized to fit in [0, 1]) to show what I am saying. I varied the random_sate, n_init but results are similar.

K means

GMM

GMM

Edit:

Regarding X and Y, these are coordinates in 2 D plane. urrently I have about 20000 X values, and 100 Y values. How I represented the data in coordinate form is by taking all possible pairs out of these.

So in this representation it is 2 dimensional, not high dimensional. But number of coordinate is large.

$\endgroup$
4
  • 1
    $\begingroup$ What is X and Y in the plot? $\endgroup$
    – Tim
    Jun 20, 2023 at 6:05
  • $\begingroup$ "the clustered scatter plot" how did you create this plot out of 100 columns? $\endgroup$ Jun 20, 2023 at 7:28
  • $\begingroup$ How did you or the algorithm decide on the number of clusters? $\endgroup$ Jun 20, 2023 at 7:28
  • $\begingroup$ For K-means by the popular elbow method. For GMM it is trial and error. I could not use density based approaches that takes epsilon distance as parameter. Those doesn't converge in reasonable time. $\endgroup$
    – pingo
    Jun 20, 2023 at 10:27

1 Answer 1

3
$\begingroup$

High-dimensional data are unfortunately inherently hard to deal with - for clustering as well as for other use cases. That low-dimensional projections or MDS show large gaps between point in the same cluster does not tell you much.

See Hayes (2011) or Bouveyon et al. (2007). You may also want to take a look at previous threads tagged both "clustering" and "high-dimensional".

$\endgroup$
4
  • $\begingroup$ Thanks but I do not thing it is high dimensional in this representation. I edited the question to clarify it. $\endgroup$
    – pingo
    Jun 20, 2023 at 10:29
  • $\begingroup$ Yes, this representation is low dimensional. But you have 100 dimensions that you cluster over, and that is where the problems come from. Or not? What else precisely do you mean by "100 Y values"? $\endgroup$ Jun 20, 2023 at 10:42
  • $\begingroup$ The 20000*100 (X, Y) pairs can be considered as location coordinates on ground. $\endgroup$
    – pingo
    Jun 20, 2023 at 11:23
  • $\begingroup$ say for k-means I did coords = numpy.vstack([X, Y[) # a (2000000,2) array Kmeans(coord, other params) $\endgroup$
    – pingo
    Jun 20, 2023 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.